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PREFACE 


This  work  was  performed  under  the  “Army  Learning”  task  for  the  Deputy  Under 
Secretary  of  Defense  for  Science  and  Technology  (DUSD  (S&T)).  The  operational  need 
for  the  work  was  identified  by  the  U.S.  Army  Training  and  Doctrine  Command 
(TRADOC).  Dr.  John  E.  Morrison  was  the  Institute  for  Defense  Analyses  (IDA)  Task 
Leader  for  the  effort.  Dr.  Kathleen  A.  Quinkert,  Chief  of  the  U.S.  Army  Research  Insti¬ 
tute  for  the  Behavioral  and  Social  Sciences’  (ARI)  TRADOC  Scientific  Coordination 
Office,  was  the  Government’s  technical  monitor. 

Mr.  Fred  Hartman  reviewed  an  earlier  draft  of  the  report  and  provided  many  use¬ 
ful  comments  that  were  incorporated  into  the  final  version.  The  cost-benefit  analysis 
employed  in  this  report  owes  much  of  its  spirit,  though  not  the  details,  to  a  resource-allo¬ 
cation  model  used  to  develop  a  plan  of  research  on  simulation-based  training  systems 
(Young,  Luster,  Stock,  Mumaw,  and  Sticha,  1986). 
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EXECUTIVE  SUMMARY 


A.  BACKGROUND  AND  REQUIREMENT 

This  document  presents  a  methodology  for  assessing  the  value  of  research  and 
development  (R&D)  opportunities  for  Army  training  and  education.  The  methodology 
uses  cost-benefit  analysis  to  determine  potential  payoffs  of  learning  science  R&D.  It  is 
applied  to  ideas  derived  from  the  Army  Science  of  Learning  Workshop  sponsored  by  the 
U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  at  the  request 
of  the  U.S.  Army  Training  and  Doctrine  Command  (TRADOC).  The  workshop  brought 
together  key  stakeholders  in  Army  training  and  education,  experts  from  academia  and 
industry,  and  representatives  of  other  Services.  The  workshop’s  participants  identified 
learning  science  findings  and  technologies  to  help  the  Army  train  Soldiers  and  grow 
leaders  for  today  and  tomorrow. 

The  workshop’s  participants  were  assigned  to  one  of  four  working  groups 
(Learning  Model,  Train  Soldiers,  Develop  Leaders,  and  Future  Capabilities)  based  on 
their  expertise.  Each  working  group  was  chaired  by  a  facilitator,  who  was  a  senior  ARI 
researcher,  and  assisted  by  a  stakeholder,  who  represented  TRADOC  interests.  The 
working  group  chair  nominated  participants  for  their  respective  groups  and  developed  the 
objectives  for  their  activities,  which  are  described  as  follows: 

•  Learning  Model.  This  group  examined  learning  models  of  other  Services 
and  organizations  to  extract  best  practices  and  lessons  learned  that  could  be 
applicable  for  an  Army  learning  model. 

•  Train  Soldiers.  This  group  examined  the  reshaping  of  the  Army’s  training 
and  education  system.  It  focused  on  how  advances  in  the  science  of  learning 
have  provided  new  pedagogical  techniques,  procedures,  and  technologies 
that  can  be  used  to  offset  pending  schoolhouse  resource  reductions. 

•  Develop  Leaders.  This  group  examined  what  we  know  about  “adaptability” 
and  explored  possible  strategies  for  accelerating  the  growth  of  key  leader 
skills  that  are  thought  to  support  adaptability.  It  also  discussed  some  of  the 
organizational  factors  (socialization,  cohesion)  that  help  establish  the  climate 
within  which  adaptable  leaders  perform. 

•  Future  Capabilities.  This  group’s  goal  was  to  envision  what  learning  sci¬ 
ence  and  emerging  technologies  will  offer  in  the  next  decade  and  to  articulate 
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how  the  Army  might  take  advantage  of  these  technologies  to  train  Soldiers 
and  grow  leaders  more  effectively. 


B.  APPROACH 

The  analysis  methodology  provides  estimates  of  the  costs  and  benefits  of  R&D 
projects  for  TRADOC  proponency.  The  intent  is  to  produce  a  generic  approach  for 
assessing  and  planning  research — not  just  for  application  to  the  current  set  of  proposals, 
but  also  for  use  with  proposals  modified  from  the  present  set  or  with  completely  new 
ones.  The  approach  is  organized  around  three  analysis  components:  proposals,  cost 
estimates,  and  benefit  estimates. 

1.  Proposals 

The  elements  of  the  current  analysis  are  21  proposed  R&D  efforts  compiled  by 
the  Institute  for  Defense  Analyses  (IDA)  based  on  some  of  the  more  important  concepts 
that  evolved  from  the  workshop’s  sessions.  Each  proposal  contains  basic  concepts  and 
estimates  of  the  required  time  and  personnel  resources. 

2.  Cost  Estimates 

The  proposal  writers  provided  cost  estimates,  which  were  decomposed  into 
implementation  costs,  maintenance  costs,  and  other  costs  (e.g.,  computer  hardware  and 
software).  Total  costs  were  calculated  in  two  ways.  First,  implementation  and  other  costs 
were  summed  to  estimate  the  first-year  (Yl)  start-up  costs.  Second,  a  more  long-term 
estimate  was  calculated  by  adding  maintenance  to  Yl  costs,  assuming  a  5-year  time 
frame  to  correspond  with  standard  Future  Years  Defense  Plan  (FYDP)  estimates. 

3.  Benefit  Estimates 

The  benefit  of  a  proposed  R&D  effort  was  conceived  as  being  analogous  to 
expected  value  (i.e.,  an  estimate  of  the  work’s  operational  impact  multiplied  by  the 
probability  of  success).  Proposal  writers  estimated  the  impact  of  each  proposed  effort, 
with  the  resulting  transformed  consensus  rating  ranging  from  10  (least  important)  to  100 
(most  important).  They  also  estimated  the  probability  of  success  principally  as  a  function 
of  the  proposal’s  technical  feasibility,  although  some  consideration  was  given  to  the 
practical  constraints  of  executing  the  research. 
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C.  RESULTS 


Benefit-to-cost  ratios  based  on  both  Y 1  and  FYDP  costs  were  calculated  for  each 
of  21  R&D  opportunuities  derived  from  the  workshop.  The  proposals  were  sorted  by  the 
ratios,  and  incremental  benefit  was  plotted  as  a  function  of  incremental  Y1  and 
incremental  FYDP  costs.  The  results  showed  that  the  method  is  sensitive  to  key  aspects 
of  cost-benefit  analysis  including  the  following  three  major  categories: 

1.  Maximize  benefits/minimize  cost.  An  optimal  point  that  balances  benefit 
and  cost. 

2.  Fixed  cost.  For  a  fixed  budget,  the  function  defines  a  set  of  R&D  proposals 
that  maximizes  total  benefit. 

3.  Fixed  benefit.  For  a  minimum  standard  of  benefit  (e.g.,  80%),  the  function 
defines  a  set  of  R&D  proposals  that  minimize  costs. 

To  illustrate  the  process,  consider  a  package  that  contains  a  single  proposal.  The 
optimal  selection  would  be  the  proposal  with  the  greatest  benefit-to-cost  ratio.  The  opti¬ 
mal  two-proposal  package  would  then  include  that  same  proposal  plus  the  one  with  the 
next  highest  benefit-to-cost  ratio.  By  ordering  the  proposals  in  this  way,  one  can  calculate 
the  total  benefit  of  the  proposal  packages  as  a  function  of  their  cumulative  costs.  More 
generally,  this  function  reveals  packages  of  proposals  that  optimize  specific  benefit  and 
cost  levels  and  that  are  sensitive  to  using  Y1  vs.  FYDP  cost  definitions.  Details  with 
extensive  examples  are  presented  in  the  document. 

D.  CONCLUSIONS 

The  principal  finding  of  this  work  is  that  application  of  cost-benefit  analyses 
produced  sensible  alternative  plans  for  a  TRADOC  R&D  program.  The  costing 
components  of  the  methodology  are  well-understood  traditional  elements  of  cost-benefit 
analyses.  The  innovative  aspect  of  the  method — the  benefits  components — had  a  notable 
role  in  its  apparent  success.  The  innovation  was  to  quantify  benefit  by  parsing  it  into  two 
subcomponents:  impact  of  the  effort  and  the  probability  of  its  success.  The  multiplicative 
combination  of  these  subcomponents  was  interpreted  as  the  “expected  impact”  of  the 
proposed  R&D  efforts. 

Given  credible  inputs,  this  document  demonstrates  how  to  integrate  R&D  propos¬ 
als  with  an  analysis  methodology  to  detennine  their  value  to  the  Anny.  With  that  in 
mind,  TRADOC  may  wish  to  alter  the  cost  or  benefit  values  and  rerun  the  analysis  on  the 
proposed  R&D  projects.  In  addition,  users  of  the  methodology  may  want  to  modify  the 
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objects  of  analysis  (i.e.,  the  R&D  proposals).  Please  note  that  perfonning  the  necessary 
judgments  and  achieving  a  consensus  of  opinion  are  themselves  nontrivial  exercises. 
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I.  INTRODUCTION 


A.  BACKGROUND 

The  U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  is  redefining  its 
role  as  “Architect  of  the  Army”  to  better  support  and  shape  the  Operational  Force.  In  that 
role,  TRADOC  is  looking  outside  traditional  organizational  relationships  to  gain  a  better 
understanding  of  the  fundamental  processes  that  underlie  learning  and  to  identify  state- 
of-the-art  technologies  that  enhance  learning. 

To  help  TRADOC  understand  these  issues,  the  U.S.  Anny  Research  Institute  for 
the  Behavioral  and  Social  Sciences  (ART)  sponsored  the  Army  Science  of  Learning 
Workshop  on  1-3  August  2006  in  Hampton,  Virginia  (Quinkert,  Morrison,  Fletcher, 
Moses,  and  Roberts,  2007).  The  workshop  brought  together  key  stakeholders  in  Army 
training  and  education,  experts  from  academia  and  industry,  and  representatives  of  other 
Services.  The  overall  purpose  of  this  workshop  was  to  identify  learning  science  findings 
and  technologies  to  help  the  Army  train  Soldiers  and  develop  leaders  for  today  and 
tomorrow. 

The  workshop’s  participants  were  assigned  to  one  of  four  working  groups 
(Learning  Model,  Train  Soldiers,  Develop  Leaders,  and  Future  Capabilities)  based  on 
their  expertise.  Each  working  group  was  chaired  by  a  facilitator,  who  was  a  senior  ARI 
researcher,  and  assisted  by  a  stakeholder,  who  represented  TRADOC  interests.  The 
working  group  chair  nominated  participants  for  their  respective  groups  and  developed  the 
objectives  for  their  activities,  which  are  described  as  follows: 

•  Learning  Model.  This  group  examined  learning  models  of  other  Services 
and  organizations  to  extract  best  practices  and  lessons  learned  that  could  be 
applicable  for  an  Army  learning  model. 

•  Train  Soldiers.  This  group  examined  the  reshaping  of  the  Army’s  training 
and  education  system.  It  focused  on  how  advances  in  the  science  of  learning 
have  provided  new  pedagogical  techniques,  procedures,  and  technologies 
that  can  be  used  to  offset  pending  schoolhouse  resource  reductions. 

•  Develop  Leaders.  This  group  examined  what  we  know  about  “adaptability” 
and  explored  possible  strategies  for  accelerating  the  growth  of  key  leader 
skills  that  are  thought  to  support  adaptability.  It  also  discussed  some  of  the 
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organizational  factors  (socialization,  cohesion)  that  help  to  establish  the  cli¬ 
mate  within  which  adaptable  leaders  perfonn. 

•  Future  Capabilities.  This  group’s  goal  was  to  envision  what  learning  sci¬ 
ence  and  emerging  technologies  will  offer  within  the  next  decade  and  articu¬ 
late  how  the  Army  might  take  advantage  of  these  technologies  to  train 
Soldiers  and  grow  leaders  more  effectively. 

B.  ORGANIZATION  AND  OBJECTIVE  OF  THIS  REPORT 

This  report  presents  specific  proposals  for  research  and  development  (R&D)  that 
were  derived  from  the  Army  workshop.  The  current  analysis  was  done  to  estimate  the 
costs  and  benefits  of  R&D  projects  for  TRADOC  proponency.  This  report  also  docu¬ 
ments  the  development  approach  for  making  programmatic  decisions  about  R&D  oppor¬ 
tunities.  The  intent  is  to  provide  a  generic  approach  for  assessing  and  planning 
research — not  just  for  application  to  the  current  set  of  proposals,  but  also  for  use  with 
proposals  modified  from  the  present  set  or  with  completely  new  ones. 
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II.  APPROACH 


The  present  analysis  is  based  on  proposals  to  exploit  learning  science  opportuni¬ 
ties  related  to  TRADOC’s  mission.  The  proposals  were  derived  from  ideas  put  forth 
during  the  Army  Science  of  Learning  Workshop.  Each  proposal  was  examined  for  its 
benefits  and  costs. 

To  maximize  the  benefits  from  the  proposals  without  regard  to  costs,  the  optimal 
plan  would  be  to  select  all  proposals.  However,  such  costs  would  be  prohibitively  expen¬ 
sive.  On  the  other  hand,  to  minimize  costs  without  regard  to  benefit,  the  optimal  plan 
would  be  to  select  the  least  expensive  proposal  (or  set  of  proposals).  However,  this 
approach  provides  no  assurance  that  the  results  would  be  of  value  to  the  Army. 

The  present  analytic  approach  represents  a  more  balanced  strategy  that  casts  costs 
and  benefits  in  a  trade  space  from  which  several  alternative  plans  can  be  derived  and 
compared.  The  intent  of  this  section  is  to  document  this  approach  in  sufficient  detail  to 
allow  it  to  be  replicated  with  alternate  proposals  and/or  a  new  set  of  estimated  values. 

A.  PROPOSALS 

The  elements  of  the  current  analysis  are  21  proposed  R&D  efforts.  The  proposals 
were  compiled  by  the  Institute  for  Defense  Analyses  (IDA)  based  on  some  of  the  more 
important  concepts  that  evolved  during  the  workshop’s  sessions.  These  proposals  were 
not  always  discussed  directly  during  the  workshop;  rather,  they  were  implied  by  ideas 
that  the  workshop’s  participants  generated. 

Each  proposal  was  given  a  long  title  to  describe  its  content  and  a  short  title 
(derived  from  the  longer  version)  to  facilitate  referring  to  these  proposals  in  the  analyses. 
Table  II- 1  summarizes  the  proposals  by  providing  the  long  and  short  versions  of  the 
titles,  the  working  group  that  generated  the  proposal,  and  a  statement  of  purpose. 

Appendix  A  provides  a  detailed  description  of  the  proposals.  Each  description 
includes  an  explanation  of  the  underlying  concept  of  the  effort  and  an  estimate  of 
required  time  and  personnel  resources.  The  order  of  proposals  in  Table  II- 1  is  the  same 
order  used  Appendix  A,  but  this  order  does  not  reflect  priority  in  cost  or  benefit. 


Table  11-1.  Summaries  of  Research  Proposals 


Long  Title 

Short  Title 

Statement  of  Purpose 

Learning  Model  Working  Group 

Selective  Implementation  of 
the  Guided  Experiential 
Learning  (GEL)  Model 

The  GEL  Model 

Apply  a  GEL  model  to  a  small  but  diverse  set  of 
training  objectives 

Implementation  of  a  Human 
Performance  Improvement 
(HPI)  Program  in  the  Army 

HPI  in  the  Army 

Study  HPI  to  determine  the  appropriate  method  for 
implementing  it  in  the  Army 

Blending  Distance  Learning 
With  Traditional  Instruction 

Blended  Learning 

Review  civilian  literature  on  blending  virtual  and  live 
training  to  interpret  application  to  constraints  of  Army 
training 

Models  To  Predict  Effective¬ 
ness  and  Efficiency  of 

Training  and  Education 

Predictive  Models 

Based  on  modern  learning  science,  develop  models  to 
predict  the  effectiveness  and  efficiency  of  training  and 
education 

Effective  and  Efficient  Knowl¬ 
edge  Acquisition,  Manage¬ 
ment,  and  Sharing 

Approaches  and  Tools 

Knowledge 

Management 

Develop  an  appropriate  architecture  for  knowledge 
management  and  supporting  tools  for  the  Army 

Life-Long  Learning  Models 

With  Intelligent  Agents  for 
Individualized  Guidance 

Life-Long  Learning 

Examine  alternative  models  for  using  intelligent  agents 
to  perform  mentoring  functions 

Train  Soldiers  Working  Group 

Systematic  Implementation 

Plan  for  Distributed  Learning 
(dL) 

Implementation 

Plan  for  dL 

Use  dL  technology  to  accelerate  training,  reduce 
costs,  reduce  personnel  requirements,  and  improve 
operational  effectiveness 

Research  and  Development 
on  Distributed  Learning  (dL) 
Topics 

R&D  on  dL 

Encourage  the  establishment  of  R&D  programs  to 
guide  the  development  and  selection  of  effective  dL 
strategies,  technologies,  and  authoring  tools 

Analyses  of  Training 

Analyses  of 

Training 

Perform  analyses  of  training  inputs  and  outputs  to 
accelerate  training  and  reduce  training  costs  and  per¬ 
sonnel  requirements 

Implementation  of  Enhanced 
Training  Capabilities 

Training 

Capabilities 

Develop  capabilities  related  to  Assignment-Oriented 
Training  (AOT)  and  for  training  in  units 

Guidelines  for  Tools  and 
Methods  To  Support  Training 

Training  Support 
Tools 

Reduce  the  “demand  side”  of  training  through 
increased  use  of  performance  and  decision  aiding 
technology  integrated  with  training  design  and 
objectives 

New  Programs  of  Research 
in  Training 

New  Training  R&D 

Encourage  R&D  programs  to  develop  a  full  systems 
approach  that  considers  costs  and  benefits  of  all 
alternatives  for  producing  human  performance 

Existing  and  Evolving  Strate¬ 
gies  for  Tailoring  Training 

Tailored  Training 

Systematically  apply  proven  techniques  for  tailoring 
training  to  needs,  prior  knowledge,  capabilities,  and 
interests  of  individual  Soldiers 
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Table  11-1.  Summaries  of  Research  Proposals  (Continued) 


Long  Title 

Short  Title 

Statement  of  Purpose 

Train  Soldiers  Working  Group  (Continued) 

Existing  and  Evolving  Strate¬ 
gies  for  Real-Time  Diagnostic 
Evaluation 

Diagnostic 

Evaluation 

Exploit  and  apply  proven  capabilities  related  to  real¬ 
time,  diagnostic  assessment  performed  using  both 
explicit  testing  and  routine  human-computer 
interactions 

Rapid  Conversion  of  Lessons 
Learned 

Lessons  Learned 

Develop  and  improve  capabilities  to  rapidly  collect, 
understand,  and  disseminate  lessons  learned  in 
theater 

Develop  Leaders  Working  Group 

Leader  Socialization 

Leader 

Socialization 

Investigate  collaborative  technologies  for  influencing 
socialization,  particularly  in  combination  with  the  Army 
Force  Generation  (ARFORGEN)  model 

Critical  Thinking  Skills 

Critical  Thinking 
Skills 

Produce  courses  that  blend  previous  experience  in 
teaching  critical  thinking  with  specific  recommenda¬ 
tions  to  improve  the  training  and  use  of  such  skills 

Experience-Based  Learning 
Methods 

Experience-Based 

Learning 

Develop  multimedia  technology  that  encourages  lead¬ 
ers  to  discover  and  learn  about  past  conflicts  from 
different  perspectives  and  analytical  approaches 

Future  Capabilities  Working  Group 

Improvisation 

Improvisation 

Directly  teach  the  art  of  improvisation  to  address  the 
need  for  “adaptive”  capabilities 

Folksonomies 

Folksonomies 

Develop  tools  that  would  allow  individuals  and  teams 
of  deployed  personnel  to  document  and  organize  their 
experiences 

Team  Task  Analysis/  Behav¬ 
ior  Modeling 

Team  Task 

Analysis 

Develop  tools  to  analyze  task  performance  from 
anthropological  models 

B.  COST  ESTIMATION 

For  each  proposal,  the  individual  who  developed  the  concept  also  provided  cost 
estimates.  These  estimates  were  decomposed  into  the  following  cost  elements: 

•  Implementation  costs.  These  are  the  costs  required  to  develop  and  imple¬ 
ment  the  intervention,  tool,  or  methods  and  are  estimated  by  the  number  of 
professional  staff-months  required. 

•  Maintenance  costs.  These  are  the  costs  required  to  maintain  the  interven¬ 
tion,  tool,  or  methods.  The  cost  metric  was  the  number  of  staff-months  per 
year. 
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•  Other  costs.  These  are  the  significant  cost  elements  required  to  implement  a 
proposal  above  and  beyond  personnel  costs.  Examples  include  specific  com¬ 
puter  hardware  or  software. 

Total  costs  were  calculated  in  two  ways.  First,  the  “implementation”  and  “other” 
costs  were  summed  to  estimate  the  first-year  (Yl)  start-up  costs.  Second,  a  more  long¬ 
term  estimate  was  calculated  by  adding  “maintenance”  costs  to  Yl  costs,  assuming  a 
5-year  time  frame  to  correspond  with  standard  Future  Years  Defense  Plan  (FYDP) 
estimates. 

C.  BENEFIT  ESTIMATION 

For  the  present  purposes,  the  benefit  of  a  proposed  R&D  effort  was  measured  by 
its  “expected  impact.”  This  value  was  conceived  as  being  analogous  to  expected  value 
and  was  defined  as  the  product  of  the  work’s  operational  impact  and  the  probability  of 
success.  The  two  component  values  of  benefits — impact  and  probability  of  success — 
were  put  into  operation  as  follows. 

1.  Impact 

The  method  for  estimating  impact  was  adopted  from  the  Simple  Multiattribute 
Rating  Technique  (SMART)  developed  by  Edwards  (1977).  Initially,  each  of  the  pro¬ 
posal  writers  independently  ranked  the  entire  set  of  proposals  from  1  (most  impact)  to 
21  (least  impact) — with  ties  in  ranks  allowed.  After  the  estimates  were  recorded,  the 
writers  then  met  face-to-face  to  resolve  differences  in  the  proposals’  rank  ordering.  Given 
the  rank  orders,  the  raters  were  tasked  to  convert  the  ranks  to  ratio  measures  by  assigning 
the  least  important  proposal  an  arbitrary  value  of  10.1  The  rater  then  considered  the  next 
least  important  proposal  and  determined  how  much  more  important  (if  at  all)  it  was  than 
the  least  important.  This  proposal  then  was  assigned  a  number  reflecting  that  ratio.  This 
process  was  continued  up  the  list,  checking  each  set  of  implied  ratios  as  each  new  judg¬ 
ment  was  made.  Thus,  if  one  proposal  was  assigned  a  weight  of  20  and  another  was 
assigned  a  weight  of  80,  it  meant  that  the  20  proposal  was  one-fourth  as  important  as  the 
80  proposal,  and  so  forth.  The  upper  ends  of  the  scale  were  allowed  to  vary,  resulting  in 
scales  ranging  from  10-40  to  10-200.  The  final  scale  was  a  weighted  average  of  the  indi¬ 
vidual  rater  scales,  which  were  transformed  so  that  the  range  of  estimated  proposal 
impact  varied  from  10  (least  important)  to  100  (most  important). 


*  According  to  Edwards  (1977),  the  starting  point  10  “...  is  used  rather  than  1  to  permit  subsequent 
judgments  to  be  finely  graded  and  nevertheless  made  in  integers”  (p.  328). 
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2.  Probability  of  Success 

Along  with  rating  the  importance  of  the  proposals,  the  proposal  writers  estimated 
the  probability  of  success,  principally  as  a  function  of  the  proposal’s  technical  feasibility, 
although  some  consideration  was  given  to  the  practical  constraints  of  executing  the 
research,  such  as  organizational  missions,  policies,  and  traditions  that  may  change  only 
with  intervention  at  the  level  of  the  Anny  Chief  of  Staff.  Like  the  impact  estimates,  raters 
independently  estimated  these  probabilities. 

D.  COMPILATION/PRESENTATION  OF  COST-BENEFIT  ESTIMATIONS 

These  initial  estimates  were  compiled  and  presented  to  all  the  proposal  writers  in 
a  face-to-face  meeting.  The  resulting  values  are  averages  that  the  proposal  writers  agreed 
captured  the  group’s  consensus. 
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III.  RESULTS 


This  section  presents  results  from  the  analyses  of  the  21  R&D  opportunities 
derived  from  the  Army  Science  of  Learning  Workshop.  These  results  refer  to  the  propos¬ 
als  by  short  titles.  Table  II- 1  lists  the  corresponding  longer  titles,  and  Appendix  A  con¬ 
tains  a  description  of  the  proposals. 

A.  COST  ANALYSIS 

Table  III- 1  summarizes  the  estimated  costs  related  to  the  21  proposed  efforts.  For 
the  staff  estimates,  note  that  most  efforts  allowed  for  parallel  work  from  multiple  staff 
members  so  that  the  chronological  time  to  develop  was  often  much  less  than  the  staffing 
estimates.  The  Y1  start-up  costs  (i.e.,  development  and  implementation)  ranged  from 
$187.5K  to  $9.4M  for  individual  efforts,  with  a  total  of  $23 .4M  for  the  entire  set  of 
21  proposals.  The  total  estimated  costs  over  a  5-year  FYDP  planning  cycle  showed  more 
variability,  ranging  from  $187.5K  to  $46. 9M,  with  a  total  of  $76. 4M  for  the  entire  set. 
These  total  potential  costs  varied  more  than  the  start-up  costs  because  they  included  the 
start-up  costs  plus  the  costs  of  maintaining  tools  and  methods  and/or  additional  related 
costs. 

B.  BENEFITS  ANALYSIS 

Table  III-2  presents  the  two  components  of  the  benefits  analysis:  the  multiplica¬ 
tive  product  of  the  probability  of  success,  /?(S),  and  its  rated  impact.  The  values  of  /?(S) 
were  generally  moderate  to  high,  ranging  from  .45  to  .85,  which  reflected  the  analysts’ 
determination  that  most  of  the  proposed  efforts  were  technically  feasible  (i.e.,  not  highly 
risky).  The  impact  ratings  ranged  from  10  to  100  and  represented  the  numerical  ratios 
among  proposals.  For  instance,  the  highest  rated  proposal  was  estimated  to  have  10  times 
the  impact  of  the  lowest  rated  proposal.  The  two  ratings  were  positively — but  not  signifi¬ 
cantly — correlated  (r  =  .35,  p  >  .11),  suggesting  that  the  judges  were  rating  the  proposals 
on  substantially  different  dimensions. 
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Table  MI-1.  Costs  Related  to  Proposals 


Short  Titles 

Personnel  Costs 
(Staff-Months) 

Total  Costs 
(K  Dollars) 

Implemen¬ 

tation 

Maintenance 
(Per  Year) 

Y1 

Costs 

FYDP 

Costs 

The  GEL  Model 

12 

6 

250 

750 

HPI  in  the  Army 

24 

0 

500 

500 

Blended  Learning 

9 

0 

188 

188 

Predictive  Models 

36 

0 

750 

750 

Knowledge  Management 

48 

12 

1,000 

2,000 

Life-Long  Learning 

48 

12 

1,000 

2,000 

Implementation  Plan  for  dL 

34 

2 

708 

875 

R&D  on  dL 

24 

24 

500 

2,500 

Analyses  of  Training 

14 

8 

292 

958 

Training  Capabilities 

18 

3 

375 

625 

Training  Support  Tools 

48 

6 

1,000 

1,500 

New  Training  R&D 

18 

18 

9,375a 

46,875a 

Tailored  Training 

12 

3 

250 

500 

Diagnostic  Evaluation 

36 

12 

750 

1,750 

Lessons  Learned 

24 

12 

500 

1,500 

Leader  Socialization 

26b 

0 

542 

542 

Critical  Thinking  Skills 

60 

4 

1,250 

1,583 

Experience-Based  Learning 

24 

1 

500 

583 

Improvisation 

21° 

9d 

438 

1,188 

Folksonomies 

36 

0 

2,250® 

8,250® 

Team  Task  Analysis 

48 

0 

1,000 

1,000 

Note  for  Table  MI-1:  Personnel  costs  based  on  $20,833  per  staff-month  (or  $250,000  per  staff- 
year).  Also,  costs  are  rounded  to  the  nearest  thousand. 

aln  addition  to  staffing  costs,  totals  include  $9M/year  budget  for  extramural  research, 
includes  additional  personnel  costs  required  to  develop  a  single  test  course. 

Clncludes  additional  personnel  costs  related  to  curriculum  development  in  first  year, 
includes  personnel  costs  related  to  follow-on  assessment, 
includes  game  development  and  hardware  costs. 
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Table  MI-2.  Benefits  Related  to  Proposals 


Short  Titles 

P(S) 

Impact 

Rating 

(10-100) 

Proportion 

Total 

Benefit 

The  GEL  Model 

0.75 

15 

0.0177 

HPI  in  the  Army 

0.45 

10 

0.0071 

Blended  Learning 

0.80 

15 

0.0189 

Predictive  Models 

0.50 

45 

0.0355 

Knowledge  Management 

0.70 

75 

0.0828 

Life-Long  Learning 

0.60 

55 

0.0520 

Implementation  Plan  for  dL 

0.85 

100 

0.1340 

R&D  on  dL 

0.65 

40 

0.0410 

Analyses  of  Training 

0.65 

65 

0.0666 

Training  Capabilities 

0.80 

35 

0.0441 

Training  Support  Tools 

0.75 

20 

0.0236 

New  Training  R&D 

0.50 

15 

0.0118 

Tailored  Training 

0.80 

90 

0.1135 

Diagnostic  Evaluation 

0.70 

85 

0.0938 

Lessons  Learned 

0.70 

60 

0.0662 

Leader  Socialization 

0.65 

20 

0.0205 

Critical  Thinking  Skills 

0.50 

55 

0.0434 

Experience-Based  Learning 

0.75 

40 

0.0473 

Improvisation 

0.55 

15 

0.0130 

Folksonomies 

0.65 

10 

0.0102 

Team  Task  Analysis 

0.80 

45 

0.0568 

The  benefit  value  for  each  proposal  was  calculated  and  then  summed  over  the 
entire  set  of  2 1  proposals.  The  total  was  used  to  determine  the  proportion  of  total  benefit 
attributed  to  each  proposed  effort,  as  shown  in  the  right-most  column  in  Table  III-2. 
These  values  for  the  individual  proposed  efforts  ranged  from  almost  1%  to  over  13%  of 
the  total  benefit  of  the  entire  set  of  proposals. 
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C.  COST-BENEFIT  TRADEOFFS 


Table  III-3  summarizes  the  values  used  to  determine  cost-benefit  tradeoffs.  For 
each  proposal,  the  benefit-to-cost  ratio  (i.e.,  benefit  per  unit  cost)  was  calculated  twice: 
for  Y1  costs  and  for  FYDP  costs.  These  ratios  were  then  used  to  rank  the  proposed  efforts 
from  1  (the  most  cost  beneficial)  to  21  (least  cost  beneficial).  The  effect  of  calculating 
benefit-to-cost  ratios  based  on  FYDP  vs.  Y1  costs  was  to  change  the  relative  rankings  of 
the  proposed  efforts.  Overall,  the  rankings  differed  by  4  or  more  positions  for  8  of  the 
2 1  proposals.  Specifically,  relatively  large  maintenance  costs  increased  total  FYDP  costs 
and  reduced  the  benefit-to-cost  rankings  of  four  proposals:  (1)  The  GEL  Model,  (2)  R&D 
on  dL,  (3)  Analyses  of  Training,  and  (4)  Lessons  Learned.  Likewise,  the  lack  or  relative 
lack  of  maintenance  costs  reduced  total  FYDP  costs  and  increased  the  relative  standings 
of  four  other  proposals:  (1)  Blended  Learning,  (2)  Predictive  Models,  (3)  Experience- 
Based  Learning,  and  (4)  Team  Tasks  Analysis. 

These  benefit-to-cost  ratios  were  then  used  to  select  optimal  subsets  or  “pack¬ 
ages”  of  proposed  efforts.  To  illustrate  this  process,  consider  a  package  that  contains  a 
single  proposal.  The  optimal  selection  would  be  the  proposal  with  the  greatest  benefit-to- 
cost  ratio.  The  optimal  two-proposal  package  would  then  include  that  same  proposal  plus 
the  one  with  the  next  highest  benefit- to-cost  ratio.  By  ordering  the  proposals  in  this  way, 
one  can  calculate  the  total  benefit  of  the  proposal  packages  as  a  function  of  their  cumula¬ 
tive  costs.  More  generally,  this  function  reveals  packages  of  proposals  that  optimize  spe¬ 
cific  benefit  and  cost  levels. 

1.  Tradeoffs  Using  Y1  (Start-up)  Costs 

To  illustrate  this  concept,  Figure  III-l  displays  the  cumulative  Y1  costs  and 
benefits  of  proposed  efforts  ordered  by  benefit-to-cost  ratios.  The  slope  starts  out  very 
steep  but  decreases  over  the  range  of  this  curvilinear  function.  Also,  as  noted  on  the 
graph,  the  last  proposal  (New  Training  R&D)  added  very  little  additional  benefit,  consid¬ 
ering  its  relatively  large  costs.  Consequently,  this  proposal  was  an  outlier  relative  to  the 
20  other  proposals  and  could  not  be  shown  in  its  actual  position  without  distorting  the 
function.  (Table  B-l  in  Appendix  B  displays  the  cumulative  Y1  costs  and  benefits  for  all 
proposals.)  The  following  discussion  shows  that  the  method  is  sensitive  to  key  aspects  of 
cost-benefit  analysis  (i.e.,  maximize  benefit/minimize  cost,  fixed  costs,  fixed  benefits). 
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Table  MI-3.  Benefit-to-Cost  Ratios 


Short  Titles 

Y1  Costs 

FYDP  Costs 

Benefit-to-Cost 

Ratio 

Rank 

Benefit-to-Cost 

Ratio 

Rank 

The  GEL  Model 

0.0450 

11 

0.0150 

15 

HPI  in  the  Army 

0.0090 

19 

0.0090 

18 

Blended  Learning 

0.0640 

7 

0.0640 

3 

Predictive  Models 

0.0300 

14 

0.0300 

9 

Knowledge  Management 

0.0525 

9 

0.0263 

11 

Life-Long  Learning 

0.0330 

13 

0.0165 

14 

Implementation  Plan  fordL 

0.1200 

3 

0.0971 

2 

R&D  on  dL 

0.0520 

10 

0.0104 

16 

Analyses  of  Training 

0.1449 

2 

0.0441 

6 

Training  Capabilities 

0.0747 

6 

0.0448 

5 

Training  Support  Tools 

0.0150 

18 

0.0100 

17 

New  Training  R&D 

0.0008 

21 

0.0002 

21 

Tailored  Training 

0.2880 

1 

0.1440 

1 

Diagnostic  Evaluation 

0.0793 

5 

0.0340 

8 

Lessons  Learned 

0.0840 

4 

0.0280 

10 

Leader  Socialization 

0.0240 

15 

0.0240 

12 

Critical  Thinking  Skills 

0.0220 

16 

0.0174 

13 

Experience-Based  Learning 

0.0600 

8 

0.0514 

4 

Improvisation 

0.0189 

17 

0.0069 

19 

Folksonomies 

0.0029 

20 

0.0008 

20 

Team  Task  Analysis 

0.0360 

12 

0.0360 

7  | 

a.  Maximize  Benefits/Minimize  Costs 

There  is  a  point  in  the  function  where  the  returns  in  benefit  begin  to  diminish 
rapidly  for  cumulative  investments.  This  point  appears  to  be  at  about  the  fifth  proposal 
(see  Figure  III-l).  Consequently,  a  package  comprising  those  five  proposals  (Tailored 
Training,  Analyses  of  Training,  Implementation  Plan  for  dL,  Lessons  Learned,  and  Diag¬ 
nostic  Evaluation)  would  provide  nearly  47%  of  the  total  benefit  of  the  entire  set  of  pro¬ 
posals  for  less  than  1 1%  of  the  total  costs  (or  $2.5M  in  Y1  costs). 
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Figure  MI-1.  Incremental  Benefit  as  a  Function  of  Cumulative  Y1  Costs 

b.  Fixed  Costs 

In  this  case,  the  funding  organization  has  a  fixed  amount  of  money  for  investment 
and  wants  to  ensure  that  it  constructs  the  optimal  package  (i.e.,  the  one  that  maximizes 
total  benefit).  The  procedure  is  to  choose  the  proposals  in  order  of  cost-benefit  ratio  until 
the  total  cumulative  cost  is  approximately  equal  to  the  fixed  amount.  Using  the  present 
data  as  an  example,  suppose  that  a  total  of  $5M  was  available  in  Y1  money.  Assuming 
that  the  total  must  not  exceed  this  budget,  the  optimal  package  would  then  comprise  the 
first  five  proposals  in  the  previous  package  plus  four  more:  Training  Capabilities, 
Blended  Learning,  Experience-Based  Learning,  and  Knowledge  Management.  This  nine- 
proposal  package  would  account  for  over  two-thirds  (67%)  of  the  total  benefit. 

c.  Fixed  Benefits 

In  this  case,  the  funding  organization  wants  to  ensure  that  a  package  of  proposals 
provides  some  stated  level  of  benefit.  The  optimal  package  in  this  case  is  the  one  that 
minimizes  cost  for  the  fixed  benefit.  For  the  sake  of  illustration,  suppose  the  funding 
organization  wanted  a  package  that  accounted  for  at  least  80%  of  the  benefit  derived 
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from  the  total  set  of  21  proposals.  The  optimal  package  (i.e.,  the  one  that  minimizes  Y1 
cost)  would  then  comprise  the  nine  proposals  specified  in  the  previous  package  plus  four 
more:  R&D  on  dL,  The  GEL  Model,  Team  Task  Analysis,  and  Life-Long  Learning.  This 
13-proposal  package,  which  accounts  for  approximately  83%  of  the  benefit,  would  cost 
$7.3M,  which  represents  about  31%  of  the  total  cost  of  all  21  proposals. 

2,  Tradeoffs  Using  FYDP  Costs 

Figure  III-2  presents  the  same  sort  of  information  as  Figure  III- 1 ,  but  this  infor¬ 
mation  is  based  on  FYDP  costs  instead  of  Y 1  costs.  As  discussed  previously,  this  dis¬ 
tinction  changed  the  rank  order  of  the  proposals  (i.e.,  considering  FYDP  costs  instead  of 
Y1  costs  provides  different  optimal  selections).  As  in  Figure  III- 1 ,  the  New  Training 
R&D  proposal  was  the  last  to  be  added  and  is  not  shown  in  its  actual  place  on  the  func¬ 
tion.  (Table  B-2  in  Appendix  B  displays  the  cumulative  FYDP  costs  and  benefits  for  all 
proposals.)  The  examples  that  follow  are  meant  to  parallel  those  described  for  Y1  costs. 


Figure  MI-2.  Incremental  Benefit  as  a  Function  of  Cumulative  FYDP  Costs 
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a.  Maximize  Benefits/Minimize  Costs 


The  point  where  the  returns  in  benefit  begin  to  diminish  rapidly  for  cumulative 
investments)  is  not  as  obvious  in  Figure  III-2  as  it  is  in  Figure  III- 1 .  However,  to  parallel 
the  previous  analysis,  the  first  five  proposals  were  chosen:  Tailored  Training,  Imple¬ 
mentation  Plan  for  dL,  Blended  Learning,  Experience-Based  Learning,  and  Training 
Capabilities.  This  package  appears  cost-beneficial  in  that  well  over  one-third  of  the  total 
benefit  (36%)  would  cost  about  $2.8M.  However,  this  cost  is  a  much  smaller  fraction  of 
the  total  cost  (about  4%)  because  the  FYDP  costs  have  a  much  larger  range  than  the  Y 1 
costs.  Also,  although  this  five-proposal  FYDP  package  roughly  parallels  the  corre¬ 
sponding  Y1  package,  the  contents  are  substantially  different.  Of  the  five  proposals  in 
both  the  Y1  and  FYDP  analyses,  only  two  (Tailored  Training  and  Implementation  Plan 
for  dL)  are  common  to  both  lists. 

b.  Fixed  Costs 

It  would  seem  reasonable  that  the  amount  of  FYDP  money  available  would  be 
greater  than  the  amount  of  Y1  money  available.  Suppose  that  $8M  was  available  across 
the  FYDP,  which  provided  the  cap  for  the  budget.  In  this  case,  the  optimal  package  (i.e., 
the  one  that  maximizes  benefit)  would  also  comprise  nine  proposals — the  five  contained 
in  the  previous  package  plus  four  more:  Analyses  of  Training,  Team  Tasks  Analysis, 
Diagnostic  Evaluation,  and  Predictive  Models.  This  package  would  consume  $7.2M  of 
the  $8M  budget  and  would  account  for  the  maximum  benefit  possible  (61%  of  the  total). 

c.  Fixed  Benefits 

Paralleling  the  Y1  package,  it  is  assumed  that  the  funder  desires  a  package  that 
accounts  for  at  least  80%  of  the  total  benefit.  The  optimal  package  (i.e.,  the  one  that 
minimizes  cost)  would  then  comprise  the  nine  proposals  specified  in  the  previous  pack¬ 
age  plus  four  more  (Lessons  Learned,  Knowledge  Management,  Leader  Socialization, 
and  Critical  Thinking  Skills).  This  13-proposal  package,  which  accounts  for  approxi¬ 
mately  82%  of  the  benefit,  would  cost  about  $12.9M. 

3.  Summary  of  Packages 

Table  III-4  summarizes  these  various  packages  of  proposals  derived  from  Y1  and 
FYDP  costs.  The  proposal  numbers  (e.g.,  1,  2,  3,  and  so  forth)  refer  to  their  relative  rank 
in  benefit-to-cost  ratios,  starting  with  the  most  cost-beneficial  proposal.  For  packages  that 
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Table  MI-4.  Optimal  Proposal  Packages  Based  on  Y1  vs.  FYDP  Cost  Estimates 


Package 

Feature 

Based  on  Y1  Costs 

Based  on  FYDP  Costs 

Proposals 

Cost 

Percent 

Total  Benefit 

Proposals 

Cost 

Percent 

Total  Benefit 

Maximize 

benefits/ 

minimize 

cost 

1.  Tailored  Training 

2.  Analyses  of  Training 

3.  Implementation  Plan  for  dL 

4.  Lessons  Learned 

5.  Diagnostic  Evaluation 

$2.5M 

47% 

1.  Tailored  Training 

2.  Implementation  Plan  for  dL 

3.  Blended  Learning 

4.  Experience-Based  Learning 

5.  Training  Capabilities 

$2.8M 

36% 

All  above  (Y1),  plus 

6.  Training  Capabilities 

7.  Blended  Learning 

8.  Experience-Based  Learning 

9.  Knowledge  Management 

S4.6M 

67% 

All  above  (FYDP),  plus 

6.  Analyses  of  Training 

7.  Team  Task  Analysis 

8.  Diagnostic  Evaluation 

9.  Predictive  Models 

$7.2M 

61% 

Fixed 

benefits 

All  above  (Y1),  plus 

10.  R&D  ondL 

11.  The  GEL  Model 

12.  Team  Task  Analysis 

13.  Life-Long  Learning 

$7.3M 

83% 

All  above  (FYDP),  plus 

10.  Lessons  Learned 

1 1 .  Knowledge  Management 

12.  Leader  Socialization 

13.  Critical  Thinking  Skills 

$12. 9M 

82% 

have  similar  features  (rows  in  the  table),  the  results  show  differences  because  of  cost 
definition  (Y1  or  FYDP).  As  the  packages  decrease  in  cost-benefit  (see  the  bottom  row  in 
particular),  the  two  definitions  produce  greater  divergence.  Using  the  Y 1  definition,  the 
bottom  row  includes  three  proposals  that  do  not  appear  on  any  selections  using  the  FYDP 
definition:  R&D  on  dL,  The  GEL  Model,  and  Life-Long  Learning.  These  results  suggest 
that  these  three  proposals  accrue  a  short-term  cost-benefit  that  is  not  reflected  in  the 
FYDP  cost  definition.  Likewise,  using  the  FYDP  definition,  the  bottom  row  contains  two 
proposals  that  do  not  appear  on  any  of  the  selections  using  the  Y1  definition:  Leader 
Socialization  and  Critical  Thinking  Skills.  These  results  suggest  that  these  two  proposals 
are  viewed  as  cost  effective  only  if  a  longer  costing  window  (5  years)  is  assumed. 
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The  observed  differences  discussed  previously  demonstrate  that  the  methods 
described  herein  (Y 1  and  FYDP)  are  sensitive  to  the  costing  method  used.2  Despite  these 
differences,  some  commonality  existed  in  proposal  selections  between  the  two  methods. 
To  provide  a  single  recommendation  that  balances  short-  and  long-term  perspectives,  a 
“compromise”  package  was  assembled.  This  package  comprised  the  top  10  proposals 
based  on  the  average  rank  for  the  two  benefit-to-cost  ratios.  Table  III-5  lists  the  resulting 
package  in  priority  order,  from  most  to  least  cost-benefit.  As  indicated,  the  total  cost  of 
this  package  over  the  FYDP  would  be  less  than  $10M.  However,  over  half  of  that  FYDP 
costs  ($5.6M)  would  be  incurred  in  Year  1. 

Table  MI-5.  Recommended  Package  of  Proposals  Based  on  Both  Cost  Estimates 


Proposals 

Y1  Cost 
($K) 

FYDP  Cost 
($K) 

1. 

Tailored  Training 

250 

500 

2. 

Implementation  Plan  for  dL 

708 

875 

3. 

Analyses  of  Training 

292 

958 

4. 

Blended  Learning 

188 

188 

5. 

Training  Capabilities 

375 

625 

6. 

Experience-Based  Learning 

500 

583 

7. 

Diagnostic  Evaluation 

750 

1,750 

8. 

Lessons  Learned 

500 

1,500 

9. 

Team  Task  Analysis 

1,000 

1,000 

10. 

Knowledge  Management 

1,000 

2,000 

Totals 

5,563 

9,979 

D.  ILLUSTRATION  OF  SELECTION  OUTCOMES 

Ultimately,  the  validity  of  the  method  is  a  function  of  the  benefit-to-cost  ratios. 
The  discussion  that  follows  documents  how  these  ratios  are  determined  and  how  the  val¬ 
ues  detennine  the  selection  of  specific  proposals.  The  intent  is  to  help  the  reader  under¬ 
stand  current  results  and  also  to  understand  the  method’s  use  for  future  applications.  Nine 


2  There  is  reason  to  believe  that  the  method  would  be  similarly  sensitive  to  the  method  for  determining 
benefit,  although  no  alternative  was  considered  for  the  present  set  of  proposals. 
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proposals  were  chosen  either  because  they  consistently  ranked  high  or  low  in  benefit-to- 
cost  ratios  or  their  rank  was  substantially  changed  because  of  the  cost  definition  used  (Y 1 
vs.  FYDP). 


1.  Proposals  Consistently  Ranked  High 

Two  proposals  (Tailored  Training  and  Implementation  Plan  for  dL)  were  consis¬ 
tently  ranked  high  in  benefit-to-cost  ratios,  regardless  of  cost  definitions.  The  two  pro¬ 
posals  were  similar  because  the  same  three  factors  accounted  for  high  rank: 

1.  Both  proposals  were  based  on  a  proven  record  of  research;  thus,  the  prob¬ 
ability  of  success,  /?(S),  was  rated  high:  .80  for  Tailored  Training  and  .85  for 
Implementation  Plan  for  dL. 

2.  Both  proposals  were  considered  highly  relevant  to  current  training  opera¬ 
tions,  and  were  thus  rated  high  for  their  potential  impact.  In  fact,  these  two 
proposals  received  the  two  highest  impact  ratings  out  of  all  21  proposals: 
100  for  Implementation  Plan  for  dL  and  90  for  Tailored  Training. 

3.  Both  proposals  incurred  some  maintenance  costs,  but  these  costs  were  rela¬ 
tively  low  (i.e.,  a  fraction  of  a  staff-year).  Their  total  FYDP  costs,  while  not 
the  lowest  of  the  entire  set,  were  relatively  low. 

2.  Proposals  Consistently  Ranked  Low 

Three  proposals  (New  Training  R&D,  Folksonomies,  and  Implementation  of  HPI) 
were  similar  because  they  were  consistently  ranked  low  in  benefit-to-cost  ratios.  How¬ 
ever,  the  reasons  for  this  result  were  different  for  the  three  proposals.  These  findings 
were  particularly  useful  because  they  revealed  conditions  under  which  proposals  are  not 
selected. 


a.  New  Training  R&D 

The  New  Training  R&D  proposal  was  rated  low  in  impact  (15  on  a  scale  of  100) 
and  low  in  probability  of  success  (.5).  In  the  defense  of  these  low  ratings,  we  believe  that 
good  research  should  be  technically  challenging  (i.e.,  low  in  probability  of  success)  and 
not  necessarily  applicable  to  operation — at  least  immediately.  However,  our  assumption 
was  that  TRADOC  was  more  interested  in  applied  (as  opposed  to  theoretical)  research, 
and  the  definitions  of  impact  and  probability  of  success  reflect  that  assumption. 

The  New  Training  R&D  proposal  was  also  the  most  costly  of  the  21  proposals, 
incurring  nearly  $47M  over  the  FYDP.  In  addition  to  a  small  staffing  requirement,  this 
total  includes  the  cost  of  the  research  itself,  estimated  to  be  $9M  per  year.  Obviously,  the 
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cost  would  be  drastically  reduced  for  the  funding  organization  if  research  funds  were 
provided  by  an  external  organization. 

b.  Folksonomies 

The  Folksonomies  proposal  was  based  on  established  anthropological  method¬ 
ology,  so  the  value  for  p(S)  was  rated  as  fairly  moderate  (.65).  Two  other  factors  were 
more  important  to  its  consistent  low  rating: 

1.  The  overall  FYDP  costs  ($8.25M)  were  second  only  to  New  Training  R&D, 
largely  because  these  costs  included  $7.5M  for  hardware  and  for  software 
development.  If  this  proposal  were  reconsidered,  the  validity  of  these  cost 
estimates  would  be  a  likely  target. 

2.  Folksonomies  (along  with  Implementation  of  HPI)  shared  the  lowest  impact 
rating  (10)  of  all  proposals.  It  was  based  on  anthropological  methods  with 
which  the  raters  were  less  than  fully  conversant.  For  this  proposal  to  be 
reconsidered,  increased  emphasis  would  have  to  be  placed  on  methodology 
and  the  description  of  potential  impact. 

c.  Implementation  of  HPI 

Ironically,  the  proposal  writers  thought  that  the  potential  impact  of  the  HPI  con¬ 
cept  (if  implemented  correctly)  was  one  of  the  largest  in  the  set.  The  problem  is  that  the 
large  potential  impact  came  at  a  high  cost — in  terms  of  financial  investment  and  disrup¬ 
tion  to  the  organization.  To  limit  cost  risks,  the  proposal  was  scoped  back  to  a  simple 
assessment  of  alternatives  for  planning  purposes.  Although  this  approach  clearly  limited 
the  investment  ($500K  over  the  entire  FYDP),  it  also  limited  the  estimated  impact  of  the 
proposal  to  one  of  the  lowest  values  in  the  set.  Furthermore,  even  though  HPI  has  worked 
well  for  the  Navy  and  the  Coast  Guard,  the  raters  were  skeptical  that  it  could  be  imple¬ 
mented  as  well  in  the  Army.  Thus,  the  probability  of  success  (.45)  was  also  markedly  low 
for  this  proposal.  An  unintentional  finding  from  this  proposal  is  that  the  method  is  very 
sensitive  to  the  scope  of  the  proposed  effort. 

3,  Proposals  Affected  by  Cost  Definitions 

The  rankings  of  four  proposals  were  especially  sensitive  to  cost  definitions,  either 
declining  or  increasing  when  total  FYDP  costs  were  considered. 

a.  Proposals  Rated  Lower  Using  FYDP  vs.  Y1  Cost  Definitions 

Two  proposals  (R&D  on  dL  and  Lessons  Learned)  substantially  declined  in 
ranking  when  total  FYDP  costs  were  considered.  For  both  proposals,  this  drop  was 
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caused  by  the  same  factor:  the  substantial  staffing  requirements  required  to  maintain  the 
proposed  work  over  all  5  years  of  the  FYDP. 

R&D  on  dL  incurs  $500K  (24  staff-months)  for  the  first  year  but  a  total  of  $2.5M 
when  the  24  staff-months  were  maintained  over  the  entire  FYDP.  As  a  result,  this  pro¬ 
posal  dropped  in  rank  from  10th  using  Y1  cost  definitions  to  16th  using  the  FYDP  defini¬ 
tions.  Similarly,  Lessons  Learned  required  24  staff-months  in  the  first  year  (i.e.,  $500K) 
but  only  12  staff-months  for  the  remaining  years  in  the  FYDP  ($1.5M  total).  As  a  result, 
this  proposal  dropped  in  rank  from  4th  using  Y1  cost  definitions  to  10th  using  the  FYDP 
definitions.  Both  proposals  illustrate  how  sensitive  this  technique  is  to  the  requirement  to 
maintain  efforts  over  the  FYDP. 

b.  Proposals  Rated  Higher  Using  FYDP  vs.  Y1  Cost  Definitions 

Two  proposals  (Predictive  Models  and  Team  Task  Analysis)  improved  in  ranking 
when  FYDP  costs  were  considered.  In  terms  of  benefit-to-cost  ratios,  Predictive  Models 
improved  from  14th  to  9th  and  Team  Task  Analysis  improved  from  12th  to  7th.  In  both 
cases,  the  reason  for  this  upward  shift  was  the  obverse  of  reason  for  the  downward  shift 
described  previously.  That  is,  both  Predictive  Models  and  Team  Task  Analysis  were  con¬ 
ceived  as  1-year  projects  with  no  follow-on  costs. 
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IV.  CONCLUSIONS 


The  overall  conclusion  from  this  study  is  that  the  application  of  cost-benefit  ana¬ 
lytic  methods  resulted  in  sensible  alternative  plans  for  a  TRADOC  R&D  program  in 
Anny  training  and  education.  The  costing  components  of  the  methodology  are  well- 
understood  traditional  elements  of  cost-benefit  analyses.  The  innovative  aspect  of  the 
method — the  benefits’  components — had  a  notable  role  in  its  apparent  success.  The  inno¬ 
vation  was  to  quantify  the  benefit  by  parsing  it  into  two  subcomponents:  impact  of  the 
effort  and  probability  of  its  success.  The  multiplicative  combination  of  these  subcompo¬ 
nents  can  be  interpreted  as  the  “expected  impact”  of  the  proposed  R&D  efforts. 

This  report  describes  the  method  in  some  detail  to  promote  its  application  using 
different  data.  For  instance,  TRADOC  may  wish  to  alter  the  cost  or  benefit  values  and 
rerun  the  analysis  on  the  proposed  R&D  projects.  For  the  present  application,  all  esti¬ 
mates  were  provided  by  the  R&D  proposal  developers.  If  TRADOC  (or  other  potential 
users)  were  to  provide  its  own  estimates,  specific  expertise  should  be  sought  to  estimate 
cost  (e.g.,  business  or  economics)  vs.  benefit  values  (e.g.,  military  technology  or  science). 
The  benefit  determination  should  also  be  decomposed  into  the  two  subcomponents: 
Military  experts  who  understand  the  effects  of  technology  on  military  operations  should 
determine  the  impact  of  a  proposed  R&D  effort,  and  scientists  who  can  judge  the  techni¬ 
cal  feasibility  of  proposals  should  estimate  the  probability  of  success.  However,  actually 
performing  such  judgments  and  achieving  a  consensus  of  opinion  are  themselves  non¬ 
trivial  exercises. 

Potential  users  may  also  want  to  modify  the  objects  of  analysis  (i.e.,  the  R&D 
proposals).  While  no  technical  impediment  exists  to  such  a  reanalysis  of  the  work,  the 
authors  of  this  report  believe  that  the  current  proposals  had  merit  before  the  analytic 
method  was  applied  (i.e.,  they  were  developed  before  the  analysis — not  to  demonstrate 
it).  As  such,  the  proposals  represent  a  synthesis  of  concepts  from  the  Army  Science  of 
Learning  Workshop,  whose  participants  included  experts  in  training  and  education  and 
key  stakeholders  in  institutional  and  unit  training  in  the  Army.  The  strength  of  this  report 
is  the  demonstration  of  how  to  integrate  credible  training  proposals  with  an  analysis 
methodology  to  determine  their  value  to  the  Anny. 
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R&D 

research  and  development 

SMART 
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TRADOC 
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I.  LEARNING  MODEL 


Working  Group  I  of  the  Army  Science  of  Learning  Workshop  examined  learning 
models  of  large-scale  organizations  outside  of  the  Anny — including  other  military  Ser¬ 
vices  (Navy  and  Coast  Guard),  universities,  and  corporations — to  extract  best  practices 
and  lessons  learned  that  could  be  applicable  for  an  Army  learning  model.  This  model  is 
descriptive  because  it  explains  basic  learning  phenomena  and  prescriptive  because  it  pro¬ 
vides  guidance  for  instructional  development  and  implementation. 

A.  SELECTIVE  IMPLEMENTATION  OF  THE  GUIDED  EXPERIENTIAL 
LEARNING  (GEL)  MODEL 

1.  Concept  and  Rationale 

The  GEL  model  is  a  structured  (i.e.,  nonconstructivist)  approach  to  instruction.  It 
has  much  in  common  with  other  structured  learning  approaches,  such  as  the  Instructional 
Systems  Design  (ISD)  model  and  the  Army’s  System  Approach  to  Training  (SAT).  The 
principal  innovation  of  GEL  is  its  integration  of  modern  cognitive  learning  theory  into 
instructional  design  and  development,  particularly  the  incorporation  of  cognitive  analysis 
techniques  into  the  design  of  instruction.  Cognitive  task  analysis  provides  a  set  of  meth¬ 
ods  for  detennining  the  knowledge  structures  and  processes  that  underlie  competent 
performance.  By  providing  prescriptions  for  imparting  those  structures  and  processes, 
cognitive  task  analysis  represents  a  method  for  developing  and  maintaining  high  levels  of 
performance. 

Some  differences  of  opinion  exist  regarding  the  appropriateness  of  GEL  to  varied 
types  of  training  content.  Some  believe  that  it  applies  to  all  types  of  training  objectives, 
whereas  others  think  that  it  is  applicable  only  to  those  objectives  that  have  substantial 
cognitive  requirements.  The  purpose  of  the  proposed  project  would  be  to  apply  the  GEL 
to  a  small  but  diverse  set  of  training  objectives.  Results  would  focus  not  only  on  metrics 
related  to  the  design  and  development  of  instruction,  but  also  on  learning  outcomes  that 
result  from  the  implementation  of  the  instruction. 
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2.  Estimated  Costs  and  Benefits 


The  principal  benefit  in  implementing  the  GEL  model  is  improved  efficiency  and 
effectiveness  of  training  processes  and  outcomes — improvement  estimates  on  the  order 
of  10%  to  20%. 

The  implementation  of  GEL  could  be  compressed  so  that  2  people  accomplish 
this  task  in  6  staff-months  (12  staff-months  in  all).  The  primary  cost  driver  is  the 
requirement  to  keep  the  GEL  program  current  with  research  and  development  (R&D) 
trends  and  to  educate  U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  training 
developers  in  its  application.  For  these  purposes,  at  least  1  person  would  be  required  to 
devote  half  his/her  time  to  this  effort  after  the  first  year  (i.e.,  6  staff-months  per  year  for 
the  remaining  4  years  of  the  FYDP). 

Implementation  of  GEL  could  result  in  avoiding  unnecessary  costs  because  of 
inappropriate  use  of  training  design  and  development  methods;  however,  this  is  benefit  is 
difficult  to  quantify. 

B.  IMPLEMENTATION  OF  A  HUMAN  PERFORMANCE  IMPROVEMENT 
(HPI)  PROGRAM  IN  THE  ARMY 

1.  Concept  and  Rationale 

HPI  is  a  concept  for  diagnosing  and  solving  problems  in  human  perfonnance.  It 
assumes  that  human  performance  problems  can  be  caused  by  a  lack  of  skills  and  knowl¬ 
edge  but  that  these  problems  can  also  be  caused  by  deficiencies  in  job  rewards,  feedback, 
equipment,  tools,  and  so  forth.  Thus,  training  is  but  one  of  many  possible  interventions 
(and  an  expensive  one  at  that)  for  improving  human  performance.  HPI  prescribes  a  care¬ 
ful  analysis  of  the  problem  to  detennine  the  appropriate  intervention. 

Although  the  concept  of  HPI  is  perfectly  reasonable,  its  implementation  in  the 
Anny  is  problematic.  It  considers  a  range  of  human  performance  problem  interventions 
that  potentially  cut  across  Army  organizations.  Training,  education,  or  doctrinal  inter¬ 
ventions  are  clearly  under  the  purview  of  TRADOC,  but  inventions  that  involve  equip¬ 
ment  modifications,  personnel  assignment,  and  other  systems  would  necessarily  involve 
non-TRADOC  organizations.  Thus,  HPI  raises  fundamental  questions  for  the  Army, 
including  the  following:  Who  should  perform  the  initial  analysis?  What  authorities 
should  he/she  possess?  Who  should  oversee  the  implementation  of  the  intervention?  Such 
questions  should  be  systematically  studied  to  determine  the  appropriate  method  for 
implementing  HPI  in  the  Army. 
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2.  Estimated  Costs  and  Benefits 


Evidence  from  other  Military  Services  and  from  private  industry  indicates  that 
HPI  implementation  can  have  a  tremendous  return  on  investment  (ROI)  in  terms  of 
(1)  improvements  in  performance  and  (2)  avoidance  of  costs  related  to  inappropriate 
performance  interventions.  However,  to  realize  such  returns,  the  Army  would  have  to 
institutionalize  the  HPI  process,  which  could  include  significant  costs.  The  proposed 
study  would  identify  and  compare  alternative  approaches  to  implementing  the  HPI  pro¬ 
cess  in  the  Anny. 

Because  the  cost  and  the  ROI  for  HPI  implementation  are  so  great,  the  present 
proposal  is  limited — it  constitutes  only  a  “plan  for  a  plan.”  This  preliminary  analysis  will 
identify  several  courses  of  action.  Assuming  appropriate  data  can  be  obtained,  a  formal 
analysis  of  alternatives  (AoA)  will  then  be  perfonned  to  determine  a  plan  of  action  that 
mitigates  risk  and  maximizes  ROI. 

The  estimated  costs  are  based  on  a  compressed  schedule  of  development  that 
involves  four  professional  analysts  who  have  expertise  in  training,  policy,  and  program 
assessment.  An  assumption  is  that  an  initial  operating  capability  (IOC)  can  be  reached  in 
6  months.  IOC  would  be  defined  as  a  workable  plan  for  implementing  HPI — not  imple¬ 
mentation  of  HPI  itself. 

C.  BLENDING  DISTANCE  LEARNING  WITH  TRADITIONAL 
INSTRUCTION 

1.  Concept  and  Rationale 

Distance  learning  (dL)  is  generally  recognized  as  a  cost-effective  alternative  to 
traditional  live  methods  of  delivering  instruction.  However,  some  training  programs  [e.g., 
the  Officer  Advanced  Course  (OAC)]  require  students  to  interact  with  instructors  and 
peers.  The  Anny  is  now  considering  how  best  to  blend  dL  with  traditional  instruction. 
Specifically,  the  Anny  asks  the  following  questions:  How  can  dL  be  combined  with  live 
instruction  to  retain  the  social  benefits  while  reducing  training  costs?  To  what  extent  can 
virtual  interaction  be  substituted  for  live  interaction? 

In  the  short  term,  such  questions  can  be  answered  by  reviewing  relevant  literature 
on  civilian  dL — literature  that  must  be  interpreted  within  the  constraints  of  Army 
training.  Nevertheless,  this  is  the  quickest  way  to  obtain  valid  advice  on  how  dL/live 
courses  can  be  blended.  This  literature  review  may  also  reveal  topics  that  require 
gathering  new  data,  and  this  would  represent  a  significant  effort.  However,  the  present 
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proposal  is  only  for  the  initial  literature  review  because  the  scope  of  any  data  gathering 
effort  cannot  be  specified  before  examining  the  literature. 

2.  Estimated  Costs  and  Benefits 

If  such  mixed  programs  were  implemented,  the  potential  savings  in  travel  costs 
and  personnel  disruption  would  be  considerable.  This  proposal  is  limited,  however, 
because  implementation  is  not  included  as  part  of  the  effort.  On  the  other  hand,  the  pro¬ 
posed  work  will  help  to  identify  ways  to  maximize  the  benefits  of  blended  learning  while 
minimizing  the  costs.  A  compressed  time  schedule  for  reaching  IOC  in  3  months  could 
be  met  by  assigning  3  staff  members  to  the  task.  IOC  is  the  delivery  of  the  literature 
review  and  appropriate  recommendations. 

D.  MODELS  TO  PREDICT  EFFECTIVENESS  AND  EFFICIENCY  OF 
TRAINING  AND  EDUCATION 

1.  Concept  and  Rationale 

The  Army  should  continue  exploring  new  learning  models,  strategies,  and  tools  to 
support  the  evolution  of  its  training  system.  In  particular,  the  Army  should  develop  mod¬ 
els  to  predict  the  effectiveness  and  efficiency  of  training  and  education.  Such  models 
should  incorporate  the  latest  in  learning  science  research  and  theories.  Bjork’s  (2007, 
1994)  reviews  of  cognition  and  memory  effects  show  how  theories  and  research  inform 
training  practices.  His  analyses  of  the  research  are  especially  important  because  they  lead 
to  concepts  that  are  not  intuitively  obvious.  For  instance,  he  shows  that  manipulations 
that  speed  the  rate  of  learning  may  not  support  long-term  retention  and  transfer.  On  the 
other  hand,  certain  manipulations  that  may  retard  learning  (“desirable  difficulties”)  may 
enhance  and  prolong  performance  and  promote  creative  transfer  of  training  to  unique 
situations.  Such  desirable  difficulties  include  spacing  rather  than  massing  study  sessions, 
interleaving  rather  than  blocking  practice  on  separate  topics,  varying  how  to-be-learned 
material  is  presented,  reducing  feedback,  and  using  tests  as  learning  events.  Some 
manipulations  (e.g.,  reducing  feedback)  may  enhance  retention  performance  at  virtually 
no  costs,  while  others  (e.g.,  varying  instructional  conditions)  may  enhance  retention  at 
the  cost  of  prolonging  original  learning.  Thus,  the  proposed  models  must  account  for 
potential  tradeoffs  between  learning  acquisition  and  retention  performance. 
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2.  Estimated  Costs  and  Benefits 


The  proposed  models  can  be  used  to  predict  the  effects  of  such  manipulations  on 
learning,  retention,  and  transfer  to  devise  optimal  strategies  for  timing,  sequencing,  and 
scheduling  the  training  and  testing  events.  To  the  extent  possible,  the  model  should  be 
derived  from  existing  research  findings,  although  original  empirical  research  may  be  pro¬ 
posed  to  fill  key  gaps.  The  work  will  require  3  staff  members  for  1  year  (36  staff- 
months).  Once  the  model  has  been  developed  and  submitted  to  TRADOC,  it  should  not 
incur  additional  maintenance  costs. 

E.  EFFECTIVE  AND  EFFICIENT  KNOWLEDGE  ACQUISITION, 
MANAGEMENT,  AND  SHARING  APPROACHES  AND  TOOLS 

1.  Concept  and  Rationale 

The  pervasiveness  of  dL  technologies  and  tools  has  promoted  a  new  approach  to 
acquiring,  managing,  and  sharing  knowledge.  This  knowledge  management  (KM) 
approach  is  embodied  in  the  Anny’s  Battle  Command  Knowledge  System  (BCKS). 
Wikipedia  (itself  a  symbol  for  the  KM  approach)  defines  BCKS  “...as  the  change  agent 
for  implementing  knowledge  management  (KM)  capabilities  into  the  training  and  mili¬ 
tary  operations  of  the  United  States  Army”  (Battle  Command  Knowledge  System  2007, 
August).  A  recent  IDA  report  (Brooks  et  ah,  2005)  analyzed  BCKS  capabilities  and  con¬ 
cluded  that  it  is  a  potentially  effective  tool  for  enhancing  learning  and  developing  situ¬ 
ational  understanding.  On  the  other  hand,  Brooks  et  al.  maintained  that  BCKS  systems 
architecture  is  not  well  integrated  and  that  BCKS’s  knowledge  acquisition  processes  is 
not  well  related  to  traditional  forms  of  instruction  and  training.  Research  is  needed  to 
develop  an  appropriate  KM  architecture  and  tools  for  the  Army.  This  architecture  should 
acknowledge  existing  technologies  (e.g.,  BCKS)  and  how  these  technologies  can  and 
should  be  integrated  to  promote  the  effective  sharing  of  actionable  knowledge.  With 
implementation  of  the  architecture,  the  Anny  may  realize  the  full  benefits  of  KM  systems 
such  as  BCKS. 

2.  Estimated  Costs  and  Benefits 

The  benefit  of  this  effort  will  occur  through  improvements  in  BCKS  and  other 
KM  capabilities.  The  architecture  work  would  be  substantial,  requiring  48  staff-months 
to  develop  and  12  staff-months  per  year  thereafter  to  maintain.  One  of  the  not-so-obvious 
maintenance  costs  would  include  the  management  of  the  culture  change  caused  by  inte¬ 
grating  KM  into  TRADOC  practices. 
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F.  LIFE-LONG  LEARNING  MODELS  WITH  INTELLIGENT  AGENTS  FOR 
INDIVIDUALIZED  GUIDANCE 

1.  Concept  and  Rationale 

Life-long  learning  views  dL  and  related  technologies  as  the  means  through  which 
education  and  training  can  be  provided  in  numerous  contexts  and  throughout  Soldiers’ 
careers.  The  contexts  go  beyond  traditional  institutional  and  unit  training  situations  and 
include  learning  opportunities  at  home,  at  the  civilian  workplace,  and  even  during  leisure 
activities  and  games.  To  provide  guidance  in  choosing  appropriate  opportunities  and  to 
assess  learning  progress,  life-long  models  include  personnel  (e.g.,  parents,  teachers,  work 
supervisors)  who  perform  an  explicit  mentoring  function.  These  mentors  provide  advice 
on  all  aspects  of  learning,  but  their  primary  training  function  is  to  prescribe  experiences 
appropriate  to  the  individual’s  stage  of  learning. 

Clearly,  one-on-one  human  mentoring  is  too  expensive  to  be  practical  on  a  large 
scale.  On  the  other  hand,  a  long  history  of  research  on  technology-based  instruction  has 
demonstrated  that  many  mentoring  functions  can  be  accomplished  through  effective  indi¬ 
vidualization  of  training.  For  instance,  Learning  Management  Systems  (LMS)  have  been 
developed  to  track  an  individual’s  progress  and  prescribe  training  exercises  appropriate 
to  this  progress.  The  challenge  is  to  expand  computer-based  mentoring  functions  beyond 
particular  training  courses  or  syllabi  and  encompass  the  whole  spectrum  of  Army  training 
and  education.  The  technology  of  intelligent  agents  provides  the  potential  to  draw  infor¬ 
mation  from  disparate  sources  and  make  integrated  learning  prescriptions  for  individual 
soldiers.  The  present  proposal  is  to  examine  alternative  models  for  using  intelligent 
agents  to  perform  such  mentoring  functions. 

2.  Estimated  Costs  and  Benefits 

A  system  of  intelligent  agents  for  mentoring  functions  is  viewed  as  an  enhance¬ 
ment  of  existing  technology-based  individualized  instruction.  The  first  step  is  to  develop 
an  intelligent  agent  model,  or  perhaps  alternative  models,  that  would  be  compatible  with 
current  distance  and  individualized  training  standards.  The  development  work  would  be 
substantial,  requiring  48  staff-months  to  execute,  but  this  work  would  be  completed 
within  1  year.  Follow-up  costs  would  be  contingent  upon  recommendations  from  the  ini¬ 
tial  research,  but  these  costs  are  estimated  to  be  12  staff-months  per  year  for  the 
remaining  4  years  in  the  FYDP. 
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II.  TRAIN  SOLDIERS 


TRADOC  continues  to  be  extraordinarily  successful  in  preparing  Soldiers  for 
operational  duty.  The  following  recommendations  are  not  aimed  at  “fixing”  anything 
TRADOC  does.  Rather,  they  are  intended  as  suggestions  to  help  TRADOC  (1)  capitalize 
on  recent  developments  and  analyses  emerging  from  the  science  and  technologies  of 
learning  and  (2)  provide  additional  training  more  efficiently  and  with  fewer  resources.  In 
addition  to  these  recommendations,  a  few  research  issues  were  included  as  opportunities 
for  consideration. 

A.  SYSTEMATIC  IMPLEMENTATION  PLAN  FOR  DISTRIBUTED 
LEARNING  (dL) 

1.  Concept  and  Rationale 

The  concept  of  this  proposed  task  is  to  use  dL  technology  to  accelerate  training, 
reduce  costs,  reduce  personnel  requirements,  and  improve  operational  effectiveness 
without  adversely  affecting  Soldiers  or  their  families.  This  goal  will  be  realized  through  a 
series  of  recommended  actions. 

a.  Budget  for  Up-Front  Costs 

Although  TRADOC  can  expect  significant  cost  savings/avoidances  by  using  dL, 
initial  investments  to  develop  materials  and  implement  dL  capabilities  are  substantial. 
Savings  through  dL  can  return  these  initial  costs  quickly — as  early  as  2  years  in  some 
analyses.  However,  this  will  require  resources  not  currently  budgeted  to  develop  and 
implement  dL. 

Obtaining  up-front  funding  for  dL  is  a  major  challenge  for  TRADOC  leadership. 
TRADOC  management  should  develop  analyses  that  show  the  expected  return  on  the 
initial  investment. 

b.  Raise  Standards  and  Expectations  for  Quality 

Most  of  the  dL  benefits  derive  from  interactivity  and  the  ability  to  tailor  instruc¬ 
tion  to  the  learner’s  needs.  Analyses  have  shown  that  these  capabilities,  in  addition  to 
reducing  per-hour  training  costs,  allow  learners  to  achieve  training  objectives  more 
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quickly  (about  two-thirds  of  the  traditional  learning  time),  thereby  freeing  learners  earlier 
for  assignment  to  duty  stations.  Analyses  have  also  shown  that  an  early  finish  reduces 
training  costs  to  a  greater  extent  than  providing  learning  at  a  distance,  which  also  results 
in  considerable  cost  savings/avoidances.  However,  using  dL  simply  as  method  by  which 
learners  can  progress  through  material  by  simply  hitting  the  “return”  key  obviates  most 
dL  benefits.  Such  training  approaches  are  becoming  increasingly  common  and  should  be 
discouraged. 

The  state  of  knowledge  shows  how  to  achieve  the  significant  benefits  promised 
by  dL.  The  state  of  practice  should  rise  to  meet  it.  TRADOC  leadership  should  insist  on 
adequate  standards  and  raise  expectations  for  dL  quality  above  the  current  norm. 

c.  Establish  a  Comprehensive  Cost  Model  With  Well-Defined  Cost 
Elements  for  dL  Training 

Cost  analyses  are  much  like  research — other  analysts  can  always  find  errors  of 
omission  and  commission.  Also,  if  TRADOC  decides  to  undertake  an  initiative  to 
develop  and  use  dL,  TRADOC  management  should  establish  a  standard  dL  cost  model, 
sufficiently  harmonized  with  other  TRADOC  cost  models,  that  would  allow  explicit  cost 
comparisons  to  be  made  for  different  training  approaches. 

d.  Establish  Routine  and  Continuous  Processes  and  Procedures  To  Ensure 
the  Quality  of  dL 

TRADOC  has  processes  and  procedures  for  assessing  and  updating  non-dL 
training,  and  TRADOC  management  should  ensure  that  these  processes  and  procedures 
are  extended  to  dL  and  sustained  by  individuals  with  dL  expertise.  Periodic  formative 
evaluations  of  cost  and  effectiveness  should  be  integrated  into  all  dL  programs.  For 
TRADOC  management  (and  perhaps  all  the  Army),  establishing  these  processes  and  pro¬ 
cedures  will  help  ensure  quality  control  when  contracting  for  dL  development. 

e.  Identify  and  Manage  the  Cultural,  Administrative,  and  Budgetary  Effects 
of  dL 

Most  TRADOC  management,  administrative,  and  budgetary  procedures  are  based 
on  instructor-led  classroom  learning,  with  quotas  and  specifically  scheduled  beginning 
and  ending  times.  These  procedures  create  what  might  be  described  as  a  TRADOC — if 
not  Anny — training  culture.  The  anytime,  anywhere  capabilities  and  processes  of  dL  are 
anomalies  for  this  culture. 
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TRADOC  leadership  will  be  required  to  incorporate  dL  into  its  training  culture. 
TRADOC  management  should  undertake  to  identify  the  specific  effects  of  dL  on  existing 
training  practices,  procedures,  and  policies  and  on  the  ability  of  TRADOC  to  incorporate 
dL  as  a  standard,  fully  accepted  training  approach.  In  this  effort,  TRADOC  management 
should 

•  Establish  incentives  and  rewards  for  early  adopters 

•  Select  materials  for  conversion  to  dL  or  development  of  dL  at  the  learning- 
module  level  rather  than  the  course  level 

•  Prepare  for  and  manage  second-  and  third-order  dL  effects 

•  Establish  processes,  procedures,  and  incentives  for  supervisors  to  use  in  dL 
training 

•  Ensure  sufficient  coordination  between  training  and  personnel  functions  so 
that  Soldiers  who  begin  and  end  programs  of  instruction  at  different  times 
receive  assignments  quickly  to  appropriate  duty  stations 

•  Establish  training  programs  for  learners,  instructors  (“train  the  trainer”),  and 
training  developers. 

2,  Estimated  Costs 

Taken  as  a  whole,  the  recommended  actions  would  require  about  34  staff-months 
to  develop  through  the  first  year  and  then  2  staff-months  a  year  to  maintain  thereafter. 
The  individual  recommendations  break  down  as  follows: 

•  Creating  a  budget  for  dL  up-front  costs  will  be  relatively  minor.  The  major 
costs  will  be  to  obtain  the  funding  required  by  the  budget.  Costs  to  create  the 
budget  and  obtain  funding  are  estimated  to  all  be  about  6  staff-months. 

•  Costs  to  devise  dL  quality  standards  will  be  relatively  minor.  The  major  costs 
will  arise  from  the  effort  and  vigilance  needed  to  have  quality  standards 
established  and  maintained.  Costs  will  be  about  2  staff-months  for  the  first 
year  and  1  staff-month  a  year  thereafter. 

•  Most  of  the  cost  modeling  effort  will  involve  reaching  consensus  about  the 
cost  elements  among  the  many  stakeholders,  including  include  how  these 
elements  should  be  defined.  Costs  to  reach  the  necessary  consensus  will  be  at 
least  14  staff-months  for  the  first  year  but  about  1  staff-month  per  year  there¬ 
after  to  maintain  and  modify  the  model. 

•  The  quality  control  of  courseware  should  require  about  1  staff-month  for 
each  dL  course.  For  costing  purposes,  we  assume  that  the  technique  is  tested 
on  5  courses  that  have  been  carefully  selected  to  represent  a  variety  of  con¬ 
tent,  audiences,  and  training  technology.  Total  costs  would  be  6  staff-months. 
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•  Identifying  and  managing  effects  of  dL  should  be  a  one-time  cost  of  6  staff- 
months. 

B.  RESEARCH  AND  DEVELOPMENT  ON  DISTRIBUTED  LEARNING  (dL) 
TOPICS 

1.  Concept  and  Rationale 

TRADOC  leadership  and  management  should  encourage  the  establishment  of 
R&D  programs  to 

•  Guide  the  development  and  selection  of  effective  dL  strategies  for  tailoring 
training  content,  sequence,  and  pace  to  the  needs,  objectives,  and  prior 
knowledge  of  individual  learners 

•  Determine  ways  to  balance  and  blend  remotely  provided  dL  and  face-to-face 
instruction 

•  Develop  and  deliver  dL  on  rugged,  handheld  devices 

•  Develop  and  provide  practicable  dL  “authoring”  (for  instructional  content) 
and  “composing”  (for  simulation  and  scenario  generation)  tools  for  use  by 
local  commanders 

•  Reduce  the  cost  of  dL  development  “authoring”  and  “composing”  through 
various  means,  including  the  development  and  use  of 

-  Sharable  instructional  objects 

-  Capabilities  that  imbue  dL  with  sufficient  “intelligence”  to  manage  and 
generate  training  interactions  on  demand  and  in  real  time. 

2.  Estimated  Costs 

Setting  up  an  R&D  effort  for  dL  capability  should  require  24  staff-months  for  Y 1 
and  24  staff-months  per  year  for  the  remaining  4  years  in  the  FYDP. 

C.  ANALYSES  OF  TRAINING 
1.  Concept  and  Rationale 

The  intent  of  this  proposal  is  to  perform  analyses  of  training  inputs  and  outputs  to 
accelerate  training  and  reduce  training  costs  and  personnel  requirements.  The  types  of 
analyses  are  described  below. 
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a.  Establish  Roles  and  Responsibilities  and  Guidelines  To  Balance 
TRADOC’s  Institutional  Responsibility  With  the  Training  Needs  of 
Operational  Forces 

TRADOC  management  should  undertake  an  initiative  in  coordination  with  the 
operational  forces  to  develop  roles  and  responsibilities  and  establish  them  as  generally 
applicable  guidelines.  A  process  to  review  these  guidelines  periodically  and  update  them 
as  needed  should  be  established  and  enforced.  At  a  more  specific  level,  TRADOC  man¬ 
agement  should  determine  the  extent  to  which  40/11  warrior  tasks  and  drills  must  be 
trained  in  Initial  Entry  Training  (IET),  assuming  the  availability  of  the  Army  Force  Gen¬ 
eration  (ARFORGEN)  program. 

b.  Establish  a  Program  To  Develop,  Identify,  and  Measure  Mission- 
Essential  Competencies  for  Army  Operations 

The  current  program  of  developing,  identifying,  and  measuring  essential  tasks  is  a 
necessary  and  major  contribution  to  Army  training  and  effective  operation.  However,  it  is 
reductionist.  Ways  are  needed  to  reassemble  tasks — essential  and  otherwise — into  opera¬ 
tional  capabilities.  The  Air  Force  has  dealt  with  this  issue  by  establishing  Mission  Essen¬ 
tial  Capabilities  (MECs).  The  Army  might  benefit  from  a  similar  approach  suitable  for  its 
operational  requirements. 

TRADOC  management  should  investigate  the  feasibility  and  value  of  establishing 
MECs  for  Army  personnel  and  collectives.  If  such  an  approach  proves  feasible  and  valu¬ 
able,  TRADOC  management  should  develop  and  implement  it. 

c.  Establish  a  Program  To  Assess  the  Military  Value  of  Training 

How  much  is  a  pound  of  training  worth?  Training  is  assumed  to  improve  the 
Army’s  operational  capabilities;  however,  the  extent  to  which  it  does  so  is  seldom  quan¬ 
tified.  Because  training  must  compete  for  resources  with  other  capabilities  whose  contri¬ 
butions  are  more  readily  quantifiable,  it  may  be  literally  and  frequently  shortchanged. 
TRADOC  management  should  undertake  analyses  to  determine,  quantitatively,  the  mili¬ 
tary  value  of  training  (i.e.,  to  determine  the  degree  to  which  different  sorts  of  training  and 
training  content  contribute  to  operational  effectiveness). 

2.  Estimated  Costs 

The  overall  cost  for  a  training  analysis  program  would  be  about  14  staff-months 
to  establish  the  program  and  8  staff-months  per  year  thereafter  to  sustain  the  program. 
Costs  for  the  individual  R&D  projects  would  be  as  follows: 
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•  The  process  for  establishing  individual  roles  and  responsibilities  should  cost 
about  12  staff-months  to  set  up  and  6  staff-months  per  year  thereafter  to 
sustain. 

•  The  program  to  develop  MECs  should  cost  about  1  staff-month  per  year 
thereafter  to  establish  and  maintain. 

•  Costs  for  assessing  the  military  value  of  training  are  especially  difficult  to 
estimate.  Our  best  estimate  is  that  it  should  cost  about  2  staff-months  to 
establish  the  program  and  1  staff-month  for  each  of  2  years  thereafter. 

D.  IMPLEMENTATION  OF  ENHANCED  TRAINING  CAPABILITIES 

1.  Concept  and  Rationale 

The  goal  of  this  proposal  is  to  develop  capabilities  to  accelerate  training  and 
reduce  training  costs  and  personnel  requirements.  Two  specific  capabilities  are  described 
below. 


a.  Provide  Assignment-Oriented  Training  (AOT)  or  Tailor  Existing 
Training  to  an  Individual’s  Current  or  Pending  Duty  Assignment 

TRADOC  leadership  should  increase  the  integration  of  personnel  and  training 
functions  to  establish  an  effective  AOT  program.  TRADOC  management  should  then 
coordinate  personnel  and  training  databases  so  that  an  individual’s  training  can  be  linked 
to  his/her  current  or  pending  duty  assignments.  To  reduce  course  completion  times  and 
costs,  this  tailoring  should,  to  some  appreciable  degree,  be  based  on  assignment,  equip¬ 
ment,  and/or  theater  requirements. 

b.  Establish  Guidelines  To  Provide  Resident  Institutional  Training  in  Units 

Various  means  (e.g.,  mobile  training  teams,  correspondence  instruction,  video- 
tele-training,  Web-based  instruction,  dL)  are  becoming  available  for  resident  institutional 
training  in  units.  TRADOC  management  should  establish  and  apply  guidelines  for  deter¬ 
mining,  on  a  case-by-case  basis,  the  most  cost-effective  approach  for  providing  training 
that  meets  specific  individual  and  unit  needs. 

2.  Estimated  Costs 

This  proposal  should  require  about  18  staff-months  to  establish  and  3  staff- 
months  per  year  thereafter  to  maintain.  The  individual  efforts  break  down  as  follows: 
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•  To  establish  a  working  relationship  between  personnel  and  training  functions 
should  require  about  12  staff-months  for  the  necessary  procedures  and 
2  staff-months  per  year  thereafter  to  maintain  them. 

•  To  provide  resident  institutional  training  in  units  should  require  about  6  staff- 
months  to  establish  the  guidelines  and  1  staff-month  per  year  thereafter  to 
maintain  and  upgrade  the  guidelines. 

E.  GUIDELINES  FOR  TOOLS  AND  METHODS  TO  SUPPORT  TRAINING 
1.  Concept  and  Rationale 

The  concept  of  supporting  tools  and  methods  is  to  reduce  the  “demand  side”  of 
training  through  increased  use  of  technology  integrated  with  training  design  and  objec¬ 
tives.  The  technologies  suggested  are  discussed  below. 

a.  Performance  and  Decision  Aids 

Training  requirements  can  be  reduced  to  some  extent  by  providing  perfonnance 
and  decision  aids,  such  as  voice-interactive  maintenance  aids,  embedded  operating 
instructions,  electronic  manuals,  and  hand-held  advisors.  TRADOC  management  should 
develop  guidelines  to  identify  and  apply 

•  The  most  cost-effective  tradeoffs  between  training  objectives  and  the  provi¬ 
sion  of  duty-site  performance  and  decision  aids 

•  Adjustments  that  should  be  made  in  training  objectives  and  programs, 
assuming  the  availability  of  performance/decision  aids 

•  The  proper  mix  of  on-site  personnel  who  will  rely  on  these  aids  and  those 
who  will  not. 

b.  Tools  for  Soldier  Self-Assessment  and  Self-Directed  Learning 

Given  the  many  capabilities  for  distributing  training  (including  but  not  limited  to 
dL),  a  Soldier  could  and  should  be  expected  to  take  more  responsibility  for  his/her  pro¬ 
fessional  growth  and  development.  However,  he/she  needs  tools  to  do  this.  TRADOC 
management  should  provide  these  tools,  which  could  be  made  available  in  various  for¬ 
mats  (e.g.,  paper,  Web,  dL  capabilities)  to  all  Soldiers.  These  tools  should  pennit  Sol¬ 
diers  sufficient  access  to  personal  and  training  resource  files  to  allow  them  to  assess  their 
needs  for  continuing  career  and  professional  growth  and  to  match  these  needs  to  the 
training  opportunities  available  to  them  from  TRADOC  or  any  other  provider. 
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c.  Continuing  Education  Units  (CEUs) 

Related  to  the  previous  proposal,  TRADOC  leadership  should  encourage  profes¬ 
sional  growth  by  requiring  all  Soldiers  to  complete  a  program  similar  to  the  CEU 
programs  offered  by  business  organizations  and  academic  institutions.  TRADOC  man¬ 
agement  should  establish 

•  CEU  equivalencies  for  TRADOC  training 

•  Sufficient  coordination  between  training  and  personnel  functions  to  ensure 
that  Soldiers  receive  credit  for  and  assignments  commensurate  with  their 
training  achievements. 

2.  Estimated  Costs 

The  various  tools  and  methods  for  supporting  training  require  about  48  staff- 
months  to  develop  and  6  staff-months  per  year  thereafter  to  maintain.  The  costs  of  the 
individual  methods  break  down  as  follows: 

•  Costs  should  be  about  12  staff-months  to  establish  the  guidelines  related  to 
performance  and  decision  aids  and  about  1  staff-month  per  year  to  maintain 
them. 

•  Costs  should  be  about  18  staff-months  to  establish  tools  for  Soldier  self- 
assessment  and  self-directed  learning  in  the  first  year  and  1  staff-month  per 
year  thereafter  to  maintain  them. 

•  The  capability  to  provide  and  administer  CEUs  should  be  about  18  staff- 
months  to  develop  in  the  first  year  and  about  4  staff-months  per  year  there¬ 
after  to  maintain  them. 

F.  NEW  PROGRAMS  OF  RESEARCH  IN  TRAINING 
1.  Concept  and  Rationale 

The  concept  of  this  proposal  is  that  TRADOC  leadership  and  management  should 
encourage  the  establishment  of  R&D  programs  to  address  a  variety  of  topics.  Although 
the  exact  topics  for  research  should  not  be  preordained,  the  following  examples  show  the 
types  of  topics  that  could  be  addressed. 

a.  Developing  a  Full  “System-of-Systems”  Approach 

Develop  a  systems-of-systems  approach  for  allocating  resources  to  training.  This 
capability  should  identify  optimal  allocations  of  resources  among  such  functions  as 
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•  Selection  and  training 

•  Personnel  job  classification  and  training 

•  Ergonomic  design  and  training 

•  Job  design  and  training. 

b.  Preparing  for  the  Unexpected 

Prepare  Soldiers  to  recognize  and  successfully  handle  unexpected  situations.  This 
work  should  address  issues  of  increasing  Soldier  adaptability  but  should  also  attend  to 
capabilities  (e.g.,  creativity,  innovativeness,  and  the  general  capability  to  “think  outside 
the  box”)  needed  for  successful  adaptations. 

c.  Documenting  Cases 

Pennit  commanders  to  record  “cases”  (stories  of  actual  military  operations)  rap¬ 
idly  and  easily  into  a  computerized  case  database,  so  that 

•  Principles  and  lessons  learned  can  automatically  be  extracted 

•  Trainers  can  rapidly  incorporate  cases  and  principles  into  programs  of 
instruction 

•  Other  commanders  can  use  these  cases  when  planning  operations. 

d.  Develop  Cognitive  Analyses 

More  fully  develop  the  processes  of  cognitive  task  analysis  and  cognitive  readi¬ 
ness  assessment  for  individuals  and  teams  so  that  these  processes  reliably  produce  the 
same  results  in  all  hands. 

e.  Improve  Performance  Measurement 

Identify  and  develop  reliable  (stable),  valid  (relevant),  and  precise  (discrimi¬ 
nating)  measures  of  individual  and  team  proficiencies  and  capabilities  from  performance 
in  simulations. 

2.  Estimated  Costs 

TRADOC  should  establish  a  small  (1-2  person)  office  to  manage  these  and  later 
training  research  projects  of  immediate  interest  and  value  (about  18  staff-months  per 
year).  The  office  should  manage  an  annual  budget  supporting  about  $9M  of  R&D  efforts 
per  year. 
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G.  EXISTING  AND  EVOLVING  STRATEGIES  FOR  TAILORING  TRAINING 


1.  Concept  and  Rationale 

The  notions  of  tailored  training  strategies,  real-time  diagnostic  evaluation,  and 
on-demand  learning  are  intimately  tied  to  dL.  Training  can  be  tailored  without  the  use  of 
technology  (e.g.,  Keller,  1968)  but  only  through  the  use  of  technology  does  it  become 
economically  feasible.  Before  the  advent  of  affordable  computer  technology,  tailored 
training  and  individualized  instruction  were  viewed  as  instructional  imperatives  but  eco¬ 
nomic  impossibilities  (Scriven,  1975;  Keller,  1985).  With  computer  technology,  tailored 
training  and  individualized  instruction  are  now  affordable  (Fletcher,  1992;  1997;  2006). 
dL  is  an  effort  to  capitalize  on  this  affordability. 

Even  without  tailored  training,  some  benefits  of  dL  remain  (e.g.,  the  ability  to 
deliver  training  and  perfonnance  aiding  anytime,  anywhere).  However,  data  show  that 
the  benefits  of  dL  are  minor  compared  to  the  training  time  and  resource  savings  to  be 
gained  from  training  tailored  to  the  needs,  abilities,  duty  assignments  and,  especially, 
prior  knowledge  of  learners  (Foster  and  Fletcher,  2003;  Dodds  and  Fletcher,  2004; 
Wisher  and  Fletcher,  2004;  Fletcher,  2006).  Savings  and  cost  avoidances  from  dL  with 
tailored  training  can  reduce  training  time  alone  by  30%  to  50%.  Such  training  would 
allow  TRADOC  to  continue  to  fulfill  its  mission  despite  downward  budget  pressures. 
Beyond  savings  and  costs,  of  course,  the  real  payoff  of  tailored  training  is  enhanced 
operational  readiness  and  effectiveness  through  ensured  human  performance  whenever 
and  wherever  it  is  needed. 

Development  of  the  techniques  to  use  computer  technology  to  tailor  instruction 
began  in  the  1950s.  By  the  early  1970s,  the  state  of  the  art  provided  reliable  techniques  to 

•  Adjust  a  learner’s  rate  of  progress,  allowing  as  much  or  as  little  time  needed 
to  reach  specified  objectives 

•  Tailor  the  content  and  the  sequence  of  instructional  content  to  each  learner’s 
or  group  of  learners’  needs 

•  Adjust  the  level  of  instruction,  making  it  as  easy  or  difficult,  specific  or 
abstract,  applied  or  theoretical,  as  necessary 

•  Adjust  to  learners’  most  efficient  learning  styles  (collaborative  or  individual, 
verbal  or  visual,  and  so  forth). 

These  capabilities  have  been  described,  discussed,  and  reviewed  by  Galanter 
(1959),  Atkinson  and  Wilson  (1969),  Suppes  and  Morningstar  (1972),  Fletcher  and 
Rockway  (1986),  and  many  others.  Unfortunately,  the  state  of  practice  in  this  area  has 
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lagged  far  behind  the  state  of  the  art.  TRADOC  should  begin  an  effort  to  apply  these 
proven  techniques  systematically. 

2.  Estimated  Costs 

Although  more  research  is  needed  and  will  prove  to  be  of  value,  the  techniques  to 
tailor  training  now  exist.  The  expense  is  in  applying  them,  and  the  scope  of  application 
will  determine  the  cost.  However,  R&D  cited  in  the  literature  suggests  that  cost  savings 
and  avoidances  realized  by  using  these  strategies  will  significantly  exceed  the  costs  of 
implementation. 

The  approach  suggested  is  to  apply  all  these  methods  to  a  single  program  of 
instruction  (POI)  that  experienced  trainers  deem  appropriate  for  tailored  training.  Initial 
start-up  costs,  which  involve  developing  and  implementing  the  POI,  are  estimated  to  be 
low  (12  staff-months)  because  the  methods  are  known.  Follow-on  work  (3  staff-months 
per  year  thereafter)  to  maintain  the  program  and  to  assess  its  effectiveness  is  required. 

H.  EXISTING  AND  EVOLVING  STRATEGIES  FOR  REAL-TIME 
DIAGNOSTIC  EVALUATION 

I.  Concept  and  Rationale 

Evaluations  of  learners  and  learner  progress  are  commonly  and  intermittently 
performed  using  explicit  tests,  instructor  assessments  or,  even,  after  action  reviews 
(AARs).  These  evaluations  are  worthwhile,  if  not  essential.  However,  management  of 
any  process,  including  learning  processes,  benefits  from  currency  and  the  ready  avail¬ 
ability  of  progress  assessment.  The  use  of  technology  in  training  makes  continuous, 
unobtrusive  assessment  of  learner  progress  possible,  and  the  use  of  tailored  training  tech¬ 
niques,  especially  those  referenced  previously,  require  such  assessment  (Fletcher,  2002). 
Just  as  with  the  one-on-one  instruction  provided  by  human  tutors,  technology  used  for 
diagnostic  evaluation  is  based  on  models  of  the  learner.  The  models  can  be  qualitative  or 
quantitative,  explicit  or  implicit,  pre-existing  or  constructed  on  the  fly,  but  they  must  be 
available  in  some  form  to  enable  technology-based  evaluations  (Bruner,  2004;  Fletcher, 
1975;  Foster  and  Fletcher,  2003). 

As  with  strategies  for  tailoring  instruction,  the  state  of  practice  in  this  area  has 
lagged  far  behind  the  state  of  the  art.  Capabilities  being  discovered  and  rediscovered  in 
areas  now  covered  by  Intelligent  Tutoring  Systems  (ITSs)  seem  particularly  promising. 
TRADOC  should  begin  a  systematic  effort  to  apply  proven  techniques  for  real-time, 
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diagnostic  assessment.  In  addition,  more  and  better  ways  to  use  the  timing,  display,  and 
computational  capabilities  of  computers  for  more  precise  diagnostic  models  of  learners 
seem  ripe  for  development.  TRADOC  should  encourage  R&D  to  exploit  and  apply  such 
capabilities  fully  through  explicit  testing  or  human-computer  interactions. 

2.  Estimated  Costs 

Although  more  research  about  diagnostic  evaluation  is  needed  and  will  prove  to 
be  of  value,  these  techniques  now  exist.  The  cost  is  in  applying  them,  and  the  scope  of 
application  will  determine  the  amount.  Initial  costs  are  estimated  at  36  staff-months  for 
the  first  year  and  12  staff-months  per  year  for  the  remaining  4  years  in  the  FYDP.  In 
addition,  the  R&D  cited  in  the  literature  suggest  that  cost  savings  and  avoidances  realized 
by  using  these  strategies  will  significantly  exceed  the  costs  to  implement  them. 

I.  RAPID  CONVERSION  OF  LESSONS  LEARNED 

1.  Concept  and  Rationale 

For  today’s  rapidly  evolving  operational  environments,  lessons  learned  in  theater 
are  of  critical  importance  in  training.  Technology  can  and  should  be  exploited  to  improve 
the  adaptability  and  agility  of  individuals,  teams,  and  units.  Capabilities  that  bring  us 
increasingly  closer  to  this  goal  are  being  developed.  Among  these  capabilities  are  those 
developed  by  the  Representing  Enriched  CONtext  (RECON)  project.  Such  projects 
enable  personnel  in  theater  to  enter  their  experiences,  or  “war  stories,”  quickly  and  easily 
into  databases  that  can  be  “understood”  and  searched  on  a  computer.  Once  these  experi¬ 
ences  have  been  entered,  the  computers,  using  techniques  developed  by  machine  learning 
research  efforts,  can  abstract  principles  from  them  for  ready  application  to  training,  per¬ 
formance  aiding,  and  decision  aiding. 

TRADOC  should  begin  a  systematic  effort  to  apply  these  capabilities  in  training 
as  a  way  to  maximize  institutional  learning  adaptability  and  improve  linkages  between 
the  generating  force  and  the  operating  force. 

2,  Estimated  Costs 

The  proposed  project  will  examine  the  various  channels  through  which  lessons 
learned  are  gathered  and  processed  and  the  types  of  infonnation  obtained  from  them. 
This  information  will  be  used  to  develop  data-packaging  methods  (e.g.,  meta  tags)  that 
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are  compatible  with  these  various  channels  and  provide  information  on  which  training 
developers  can  act. 

The  Y 1  start-up  costs  (24  staff-months)  cover  the  effort  to  develop  these  methods, 
whereas  the  follow-on  funding  (12  staff-months  per  year  for  the  remaining  4  years  in  the 
FYDP)  is  for  implementing  the  methods  in  an  operational  environment. 
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III.  DEVELOP  LEADERS 


For  several  years,  the  U.S.  Army  has  recognized  the  need  to  develop  self-aware, 
adaptive  leaders  who  can  perform  effectively  in  a  broad  range  of  situations  across  the  full 
spectrum  of  operations.  This  working  group  examined  what  we  know  about  adaptability 
and  explored  possible  strategies  for  accelerating  the  growth  of  key  leader  skills  that  are 
thought  to  support  adaptability.  The  group  also  discussed  some  of  the  organizational 
factors  (socialization,  cohesion)  that  help  to  establish  the  climate  within  which  adaptable 
leaders  perfonn. 

A.  LEADER  SOCIALIZATION 
1.  Concept  and  Rationale 

A  recent  study  about  the  development  of  Army  leaders  says  that  leaders  should 
“...create  a  command  climate  that  supports  operational  excellence  and  also  motivates 
competent  people  to  continue  their  military  service”  (Ulmer,  Shaler,  Bullis,  DiClemente, 
and  Jacobs,  2004).  To  create  that  climate,  this  study  highlights  two  leader  behaviors 
involving  socialization  skills  of  major  significance: 

1 .  Gets  out  of  the  headquarters  and  visits  the  troops 

2.  Builds  and  supports  teamwork  within  staff  and  among  units. 

Officers  who  participated  and  responded  included  the  Commanders,  Assistant 
Division  Commanders,  Chiefs  of  Staff,  8  members  of  each  Division  Staff,  and  from  6  to 
10  subordinate  commanders  in  each  Division.  They  rated  the  socialization  items  above  as 
“most  important  for  setting  climate”  and  “behaviors  for  Division  Commanders  to  work 
on.”  The  question  here  is  how  such  socialization  skills  can  be  enhanced  in  officer  career 
development. 

Leader  socialization  with  troops  is  different  from  Soldier  socialization  in  peer 
groups.  Peer  groups  have  day-to-day  activities  and  participation  in  classes  and  in  units 
for  getting  acquainted  and  developing  cohesion.  In  contrast,  leaders  need  to  make  them¬ 
selves  available  to  their  troops  to  foster  respectful  socialization,  and  that  challenge  is 
increasingly  difficult  because  of  the  emphasis  on  distributed  operations,  electronic  net¬ 
works,  and  larger  (remote)  areas  of  responsibility.  One  way  to  proceed  is  to  take 
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advantage  of  electronic  communication  as  a  means  of  helping  instead  of  hindering  the 
socialization  process. 

Research  with  peer  groups  (Blanch,  Orvis,  and  Wisher,  2003)  shows  that  Web- 
based  communication  before  face-to-face  meetings  can  accelerate  the  socialization  pro¬ 
cess.  The  technique  was  demonstrated  in  a  trial  application  for  reserve  component  offi¬ 
cers  enrolled  in  the  Annor  OAC.  It  worked  by  using  text  messaging  during  a  block  of 
instruction  before  resident  coursework.  A  similar  technique  should  be  explored  to  deter¬ 
mine  the  benefits  provided  to  Commanders  who  routinely  exchange  infonnation  with 
their  Soldiers.  These  should  be  day-to-day  communications  and  not  only  “special  mes¬ 
sages  from  your  Commander.”  Topics  (e.g.,  updates  on  daily  activities  and  staff 
meetings)  should  encourage  similar  infonnation  exchanges  from  the  Soldiers.  These 
exchanges  would  provide  a  foundation  for  more  meaningful  face-to-face  visits.  Collabo¬ 
rative  technologies  (e.g.,  text  messaging)  for  influencing  socialization  wanant  further 
investigation,  particularly  in  combination  with  the  ARFORGEN  model  that  includes  a 
form  of  force  stabilization.  Force  stabilization  emphasizes  Leader  socialization  with  the 
Soldiers  as  an  important  way  ensure  a  good  long-tenn  working  relationship. 

2.  Estimated  Costs 

The  cost  of  this  study  is  estimated  at  about  18  staff-months  plus  costs  of  imple¬ 
mentation  per  course  or  application,  estimated  at  8  staff-months.  For  costing  purposes,  a 
single  test  course  is  assumed  to  be  developed  for  testing  purposes.  The  study  involves 
two  phases:  (1)  assess  the  benefits  of  collaborative  technologies  for  classroom  or  Com¬ 
mander-unit  socialization  and  (2)  verify  its  utility  through  interviews  with  officers  who 
had  the  socialization  experience  and  those  who  did  not.  Implementation  would  incur  the 
cost  of  configuring  networks  for  day-to-day  within-group  communications,  designing 
computer-based  introductions  to  new  assignments  or  courses,  and  combining  these  with 
collaborative  technologies. 

Costs  pertain  to  coordinators  required  to  configure  a  specific  socialization  capa¬ 
bility  and  to  educate  users.  This  cost  would  be  applicable  for  each  implementation.  If 
successful,  the  Anny  could,  with  little  to  no  added  expense,  introduce  the  approach 
broadly  using  available  computer  equipment. 
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B.  CRITICAL  THINKING  SKILLS 


1.  Concept  and  Rationale 

Ample  evidence  exists  that  critical  thinking  skills  (CTS)  have  value  for  battle 
command  and  tactical  decision  making,  particularly  when  uncertainty  and  time  pressure 
are  factors.  At  the  same  time,  according  to  an  Army  workshop  report,  Training  Critical 
Thinking  Skills  for  Battle  Command  (Reidel,  Morath,  and  McGonigle,  2001),  little  con¬ 
sensus  exists  on  what  exactly  constitutes  critical  thinking  and  how  it  should  be  measured. 
This  proposal  suggests  that  the  Army  intensify  its  efforts  to  implement  training  of  critical 
thinking  based  on  experience  gained  previously. 

In  general,  we  can  characterize  critical  thinking  as  an  ability  to  assess  one’s  own 
thinking  and  the  thinking  of  others  to  gain  understanding  and  achieve  wise(r)  judgments. 
Ample  descriptions  about  critical  thinking’s  concepts,  values,  and  principles  to  help 
inform  our  learning  strategies  supplement  this  broad  definition.  More  difficult,  however, 
is  the  development  of  good  methods  and  procedures  for  learning  and  perfonning  critical 
thinking. 

A  plausible  strategy  for  developing  CTS  combines  infonnation-based  instruction 
on  the  concepts  (knowledge),  demonstration  of  the  processes  (performance-based),  and 
guided  practice  (coaching/feedback)  with  realistic  problems.  Yet,  the  primary  implemen¬ 
tation  infonnation  (e.g.,  Cannon-Bowers  and  Salas,  1998;  Gerras,  2006;  and  Reidel  et  al., 
2001)  is  mostly  concerned  with  theories  and  general  procedures  for  how  to  develop  criti¬ 
cal  thinking.  Far  fewer  specifics  are  available  about  what  to  do  and  what  works. 

An  Army  example  of  good  instructional  principles  is  the  Think  Like  A  Com¬ 
mander  (TLAC)  vignette  program  to  train  adaptive  leaders  from  the  Brigade  Commander 
down.  It  presents  a  simulation  of  a  situation  to  a  commander  in  the  context  of  a  general 
problem  that  he  has  experienced  previously  but  with  a  different  specific-problem  situa¬ 
tion.  The  commander  works  through  his  interpretation,  his  reasoning,  and  the  types  of 
possible  actions.  Information  is  provided  to  relate  that  thinking  back  to  the  problem¬ 
solving  thinking  processes  of  great  commanders.  Through  a  series  of  vignettes,  a  com¬ 
mander  starts  to  think  more  about  the  process  of  resolving  problems  while  he  practices 
critical  thinking. 

Using  the  same  kind  of  teaching  strategy  represented  by  TLAC,  the  Anny  has 
begun  adding  more  capability  to  foster  critical  thinking.  An  initial  implementation  has 
begun  at  the  U.S.  Military  Academy,  West  Point.  In  addition,  several  Anny  schools — 
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Army  Management  Staff  College  (AMSC),  Command  and  General  Staff  College 
(CGSC),  Anny  War  College  (AWC) — have  explored  various  curricula  for  critical 
thinking.  The  Center  for  Army  Leadership  (CAL)  has  a  current  effort  developing  meas¬ 
ures.  A  recent  effort  just  completed  by  the  U.S.  Anny  Research  Institute  for  the  Behav¬ 
ioral  and  Social  Sciences  (ARI)  uses  Web-based  training  to  focus  on  eight  CTS.  Each 
skill  module  is  2  hours  long  although  longer  versions  (8  to  10  hours)  for  four  of  the  skills 
are  available.  This  product  is  ready  to  deliver  now. 

Teaching  critical  thinking  is  expanding  in  the  Army.  However,  four  recommen¬ 
dations  (about  what  to  teach)  presented  originally  in  the  Workshop  about  training  critical 
thinking  (Reidel  et  al.,  2001)  need  added  attention: 

1.  Train  interpersonal  skills  for  the  application  of  critical  thinking.  One  of 

the  barriers  to  the  use  of  critical  thinking  is  social  awkwardness.  Learning 
about  critical  thinking  is  important,  but  its  use  will  be  limited  unless  others 
accept  it  and  one’s  command  supports  it. 

2.  Model  the  use  of  critical  thinking  skills.  Training  should  immerse  students 
in  the  critical  thinking  process,  either  through  working  with  someone  who  is 
critically  thinking  or  by  showing  them  how  to  work  through  the  problem 
using  a  set  of  principles.  Thus,  facilitators  must  be  good  critical  thinkers  and 
receive  training  and  feedback  in  critical  thinking. 

3.  Train  for  transfer  of  training.  Even  using  simulations,  all  possible  situa¬ 
tions  cannot  be  simulated.  Methods  for  facilitating  transfer  of  training  to 
other  situations  are  available  and  should  be  used  when  developing  critical 
thinking  training  programs. 

4.  Include  critical  thinking  training  in  every  course  taught  in  the  Army. 

This  can  be  accomplished  easily  on  one  level  by  emphasizing  elements  (e.g., 
asking  open-ended  questions  and  reinforcing  the  concept  of  working  through 
problems)  that  promote  critical  thinking.  Far  more  difficult  is  training  some¬ 
one  to  deal  with  novel  situations  through  the  use  CTS  and  interactive  exer¬ 
cises  as  part  of  the  course  design. 

Overall,  the  Anny  seems  to  be  acquiring  good  experience  in  teaching  critical 
thinking  and  should  increase  efforts  to  offer  more  courses.  For  maximum  effect,  the 
Anny  should  blend  its  past  experience  with  these  four  recommendations  to  improve  the 
teaching  and  the  use  of  critical  thinking. 

2.  Estimated  Costs 

To  teach  critical  thinking  systematically,  the  cost  of  this  project  would  be  on  the 
order  of  24  staff-months  for  each  new  course  preparation.  Augmenting  a  course  with 
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critical  thinking  exercises  should  be  possible  for  one-quarter  to  one -half  that  amount.  The 
costing  projections  assume  that  a  minimum  of  two  new  courses  will  be  developed  and 
that  two  courses  will  be  augmented.  Maintaining  each  course  would  require  about  1  staff- 
month  per  year  thereafter.  The  benefit  would  be  improved  efficiency  and  effectiveness 
not  just  for  training  of  CTS,  but  also  for  solving  problems  on  the  asymmetric  battlefield 
and  in  stability  operations. 

C.  EXPERIENCE-BASED  LEARNING  METHODS 

1.  Concept  and  Rationale 

The  study  of  past  conflicts  and  their  successes  and  problems  is  an  integral  part  of 
learning  leader  skills.  Leaders  read  and  discuss  accounts  of  battle  strategies  and  tactics, 
illustrating  points  with  the  help  of  situation  maps,  AARs,  photos,  and  video.  Personal 
accounts  from  veterans  sometimes  add  color  and  detail.  A  newer  development  is  the  use 
of  multimedia  technology  to  enhance  the  experience,  illustrated  with  examples  from  the 
Mazar-e  Sharif  campaign  in  Afghanistan  in  2001  (Knarr  and  Richbourg,  2007). 

The  concept  is  to  develop  digital  versatile  discs  (DVDs)  with  data,  analyses,  and 
lessons  learned.  However,  the  DVD’s  built-in  capability  allows  much  more  than  that.  The 
DVD  is  set  up  to  be  conducive  to  discovery  or  directed  learning  at  multiple  levels  and 
from  different  perspectives  and  analytical  approaches.  For  Mazar-e  Sharif,  the  material  is 
said  to  be  robust  enough  to  support  nine  topics,  as  follows: 

1.  Traditional  (as  threads  develop  during  research) 

2.  Levels  of  war  (tactical,  operational,  strategic) 

3.  Principles  of  war  (Mass,  Objective,  Offense,  Surprise,  Economy  of  Force, 
Maneuver,  Unity  of  Command,  Simplicity) 

4.  Doctrine,  Organization,  Training,  Material,  Leadership,  Personnel,  Facilities 
(DOTMLPF) 

5.  Diplomatic,  Infonnation,  Military,  Economic  (DIME) 

6.  Mission,  Enemy,  Terrain  and  weather,  Troops  -  Time,  Civil  (METT-TC) 
considerations  to  include  cultural  aspects 

7.  Hypothesis  testing 

8.  Case  study 

9.  Staff  rides. 
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For  example,  the  second  topic — levels  of  war — can  be  used  to  analyze  various 
campaign  aspects:  political,  cultural,  alliances,  information  operations,  interdependen¬ 
cies,  rules  of  engagement  (ROE),  logistics,  and  adaptability.  The  approach  allows  drilling 
into  and  developing  a  subtopic  such  as  ROE,  which  specify  the  circumstances  and  limi¬ 
tations  under  which  forces  conduct  operations  other  than  war  (OOTW)  or  begin  or  con¬ 
tinue  combat.  The  index  for  the  Mazar-e  Sharif  example  points  to  several  video  clips, 
audio  clips,  and  transcripts  that  specifically  address  or  relate  to  ROE.  One  discussion  is 
about  the  problems  associated  with  positively  identifying  targets  from  aircraft  at 
20,000  feet.  The  lead-in  to  that  is  “Boots  on  the  Ground,”  where  the  Secretary  of  Defense 
explains  the  need  for  ground  troops:  “In  Afghanistan,  precision-guided  bombs  from  the 
sky  did  not  achieve  their  effectiveness  until  we  had  boots  and  eyes  on  the  ground  to  tell 
the  bombers  exactly  where  to  aim.”  Another  discussion  by  General  Franks  is  about  “How 
well  ROE  supported  the  targeting  process”  and  “How  he  would  consider  changing  the 
ROE  development  process  to  support  the  mission.”  Such  information,  combined  with 
battle  re-creations  using  computer-generated  virtual  worlds,  provides  a  powerful  educa¬ 
tional  tool  for  senior-level  conferences  and  leader  development  courses. 

The  DVD  capability  to  learn  from  experience  is  a  technology-rich  way  to  develop 
leader  skills  based  on  past  conflicts.  Multimedia  battle  re-creations,  expert  commentaries, 
and  supporting  materials  allow  leaders  to  study  and  analyze  near-real  combat  conditions 
from  command,  Soldier,  and  enemy  perspectives.  It  is  the  next  best  thing  to  being  there. 
The  Army  should  take  advantage  of  not  only  the  materials  on  Mazar-e-Sharif,  which  is 
ready  for  use,  but  should  also  develop  further  two  other  such  re-creations:  the  Battle  of 
73  Easting  from  the  1991  Gulf  War  and  the  Battle  for  Fallujah,  Iraq,  in  2004. 

2.  Estimated  Costs 

This  implementation  is  possible  for  the  cost — estimated  at  24  staff-months — of 
developing  teaching  points  emphasizing  specific  themes  (e.g.,  cultural  sensitivity;  adapt¬ 
ability)  to  accompany  a  multimedia  battle  re-creation  and  about  3  to  6  staff-months  to 
IOC.  Maintenance  costs  would  be  about  1  staff-month  per  year  thereafter.  Teaching 
development  should  focus  on  the  needs  of  particular  schools  and  be  done  in  cooperation 
with  their  subject  matter  experts  (SMEs). 
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IV.  FUTURE  CAPABILITIES 


Members  in  this  working  group  envisioned  what  learning  science  and  emerging 
technologies  will  offer  within  the  next  decade  and  articulated  how  the  Army  might  train 
Soldiers  and  develop  leaders  more  effectively. 

A.  IMPROVISATION 

1.  Concept  and  Rationale 

During  the  conference,  much  discussion  centered  on  developing  “adaptive”  lead¬ 
ers,  without  much  being  offered  to  specify  what  that  means.  At  the  same  time,  one  par¬ 
ticipant  (Wayne  Hodgins)  brought  up  the  (undeveloped)  notion  of  “strategic  forgetting.” 
What,  he  wondered,  might  it  mean  if  unlearning  past  norms  of  behavior  became  useful  in 
the  future ?  This  question  rekindled  the  story  of  an  accomplished  pianist  who  was  having 
a  terrible  time  learning  a  new,  radical  piece  of  music  because  it  required  that  she  get  off 
the  piano  stool,  stand  up,  walk  around  to  the  side  of  the  piano  and  strike  the  strings 
directly  with  her  fists.  Maybe  unlearning  “norms”  is  worth  investigating. 

Most  of  the  literature  on  improvisation  seems  to  focus  on  musical  or  theatrical 
performance,  both  of  which  require  massive  content  domain  knowledge  before  the  prac¬ 
tice  of  improvisation  can  begin.  This  did  not  seem  helpful,  as  we  are  advised  that  adapta¬ 
bility  skills  in  the  Army  have  to  be  pushed  down  the  ranks  to  less-experienced  personnel. 
Researching  “instruction  to  develop  adaptive  leaders”  yields  lots  of  infonnation  in  Mas¬ 
ter  of  Business  Administration  (MBA)  programs  and  in  educational  (i.e.,  school)  leader¬ 
ship  programs,  but,  again,  these  appear  to  be  focused  on  extending  traditionally  fixed 
bounds  of  knowledge  along  historically  established  domains. 

Perhaps  more  interesting  is  research  done  at  the  edges  of  experience,  such  as  that 
conducted  with  individuals  who  have  autism  (Marckel,  Neef,  and  Ferreri,  2006): 

Language  and  communication  are  major  areas  of  concern  for  children  with 
autism  ( Diagnostic  and  Statistical  Manual  of  Mental  Disorders,  American 
Psychiatric  Association,  2000).  Research  has  shown  that  augmentative  and 
alternative  communication  (AAC)  systems,  such  as  sign  language,  elec¬ 
tronic  communication  aids,  and  the  picture  exchange  communication  sys¬ 
tem  (PECS),  can  increase  the  communicative  interactions  of  children  with 
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autism  and  enable  them  to  exercise  control  over  their  environments  (e.g., 
by  making  requests). . . 

Fields  at  less-certain,  farther  edges  of  practice  such  as  this  [e.g.,  stranded  moun¬ 
tain  climbers,  emergency  room  physicians,  and  National  Aeronautics  and  Space  Admini¬ 
stration  (NASA)  astronauts]  may  offer  more  promising  leads.  The  Wharton  School  at  the 
University  of  Pennsylvania  has  conducted  some  research  in  this  area,  defining  a  concept 
of  “resiliency”  as  being  particularly  pertinent. 

Directly  teaching  the  art  of  “improvisation”  might  be  one  way  of  addressing  the 
needed  “adaptive”  capabilities  in  a  purposeful  manner,  rather  than  in  an  indirect  manner. 

2.  Estimated  Costs 

Several  cohorts  would  have  to  participate  in  a  candidate  training  program  that 
included  individual  aptitude,  capability,  and  personality  tests.  The  estimated  level  of 
effort  for  this  work  would  be  about  9  staff-months.  Follow-on  field  studies  would  also  be 
needed.  A  curriculum  might  be  developed  for  as  little  as  12  staff-months.  Testing  it 
would  take  a  longer  time,  but  no  real  benefit  could  be  realized  without  such  assessments. 
Testing  time  needed  would  be  at  least  36  staff-months.  The  benefits  are  hard  to  project, 
but  this  seems  to  be  something  everyone  wants — without  much  science  to  support  the  oft- 
stated  desires  or  to  point  to  productive  new  directions. 

B.  FOLKSONOMIES 

1.  Concept  and  Rationale 

Troops  that  benefit  from  the  “lessons  learned”  of  their  predecessors  are  far  more 
likely  to  survive  than  those  who  do  not.  Technologies  that  can  support  the  capture,  repre¬ 
sentation,  and  accessibility  of  “lessons  learned”  will  be  of  great  use  to  newly  deployed 
Soldiers.  Traditional  knowledge-capture  and  knowledge-representation  technologies  have 
emphasized  brittle  technologies  that  can  be  exploited  only  by  research  scientists.  These 
technologies  are  of  no  use  to  individuals  who  do  not  have  highly  specialized  educations 
(i.e.,  enlisted  personnel).  New  forms  of  knowledge-capture  technologies  offer  greater 
promise  in  the  form  of  “folksonomies.”  Reporting  in  Wired.com,  Terdiman  (2005) 
described  these  technologies  as  follows: 

These  days,  a  growing  number  of  sites  whose  content  is  user-created  rely 
on  tagging  systems,  also  known  as  folksonomies,  for  the  added  value 
Butterfield  is  talking  about.  Flickr  and  the  social-bookmarking  site 
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Delicious,  along  with  Furl,  are  generally  considered  folksonomy  trail- 
blazers,  but  now  sites  like  MetaFilter  and  the  blog  index  Technorati  have 
jumped  on  board,  and  more  are  expected  to  follow. 

“It’s  very  much  people  tagging  infonnation  so  that  they  can  come  back  to 
it  themselves  or  so  that  others  with  the  same  vocabulary  can  find  it,”  said 
Thomas  Vander  Wal,  the  information  architect  credited  with  coining  the 
term  ‘folksonomy.’ 

“To  me,  they’re  a  great  new  organization  tool  for  applications  and  large 
content  sites,”  said  Matt  Haughey,  the  founder  of  MetaFilter.  “Tags  are 
great  because  you  throw  caution  to  the  wind,  forget  about  whittling  down 
everything  into  a  distinct  set  of  categories  and  instead  let  folks  loose  cate¬ 
gorizing  their  own  stuff  on  their  own  terms.  It  hasn’t  always  been  easy  to 
get  users  to  take  on  such  responsibility.  But  as  more  people  understand 
what  tags  are,  how  they  work,  and  why  they’re  important,  the  number  of 
participants  in  folksonomies  has  grown.” 

“Tags  are  great,  but  the  thing  that  is  hard  is  getting  people  to  use  them,” 
said  Caterina  Fake,  who  co-founded  Flickr  with  husband  Stewart 
Butterfield.  “But  the  thing  that  has  happened  recently  is  they’ve  become 
part  of  a  social  arena  in  which  they  are  valuable  not  only  to  the  individual 
but  to  the  group.” 

The  Armed  Services  excel  at  developing  identity  communities.  One  could  argue, 
however,  that  these  are  communities  established  by  an  assault  on  individuality  and  that 
this  has  negative  consequences  for  future  (creative)  behavior.  Much  information  is  avail¬ 
able  to  recommend  the  establishment  of  team  vs.  individual  self-interest  but  as  much 
information  may  be  available  to  recommend  the  establishment  of  a  sense  of  team  created 
by  professional  capacities  that  are  different  from  those  officially  recognized  [i.e.,  in  ways 
not  accommodated  by  traditional  military  occupation  specialties  (MOSs)  and  other  mili¬ 
tary  distinctions]  and  in  ways  that  might  be  not  possible  to  predict. 

Tools  that  would  allow  individuals  and  teams  of  deployed  personnel  to  document 
and  organize  their  experiences,  replete  with  anomalies,  personal  reflection,  speculation, 
and  advisory  commentary  in  their  own  language,  would  be  valuable  to  newly  deployed 
personnel  trying  to  keep  pace  with  ever-changing  conditions  on  the  ground.  Newly 
deployed  troops  have  little  difficulty  learning  the  “lingo”  of  the  field,  but  they  may  have 
great  difficulty  learning  the  meaning  and  nuance  of  the  language. 

Ideally,  data  captured  from  the  field  would  be  fed  back  to  residential  training 
schools,  where  students  would  engage  the  content  in  high-fidelity  training  “games.”  The 
high-end  capacity  of  computer  servers  at  the  residential  training  schools  would  provide 
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for  realistic  scenarios,  imparting  a  high  degree  of  engagement  through  the  luddologic 
aspect  of  game-play  behavior  and  the  consequent,  narratologic  sense  of  self-agency  that 
is  realized  in  virtual  environments  with  highly  probable  consequences.3  This  game  envi¬ 
ronment  would  teach  the  skills  required  by  on-line  community  development,  based  on 
situational  needs.  Fielded,  hand-held  tools  could  maintain  the  task  development  activities 
of  data  capture  and  taxonomy  development  for  real-time  use  and  would  recall  benefits  of 
the  narratologic  aspect  of  the  residential  game-play  environment. 

2.  Estimated  Costs 

An  anthropological  study  of  practices  in  use  in  the  field  (to  be  institutionalized  in 
technology)  could  be  accomplished  for  36  staff-months.  Game  development  costs  could 
range  from  $3M  to  $9M.  Development  of  software  for  commercial  off-the-shelf  (COTS) 
hand-held  devices  for  knowledge  capture  and  representation  could  cost  $1M  to  $2M. 

C.  TEAM  TASK  ANALYSIS/BEHAVIOR  MODELING 

1.  Concept  and  Rationale 

Much  has  been  done  for  team  or  group  task  analysis.  The  Department  of  Defense 
(DoD)  has  paid  for  most  of  it.  ARI  has  conducted  much  research  in  this  area,  but  most  of 
what  has  resulted  appears  to  be  brittle  and  of  little  help  in  being  predictive. 

Research  that  looks  to  be  effective  in  modeling  crowd  behavior  under  various  cir¬ 
cumstances  is  ongoing.  The  reverse  of  the  statistical  Simpson  effect4  may  be  at  work. 


3  Luddologic  and  narratologic:  Narratology  is  the  study  of  narrative  and  narrative  structure  and  the  ways 
they  affect  our  perception.  The  narrativists’  approach  video  games  in  the  context  of  what  Janet  Murray 
(Professor  at  Georgia  Tech)  calls  “Cyberdrama.”  The  narrativists’  major  concern  is  with  video  games 
as  a  storytelling  medium,  one  that  arises  out  of  interactive  fiction.  The  ludologists  break  sharply  from 
this  view.  Their  perspective  is  that  a  video  game  is  first  and  foremost  a  game  and  that  it  needs  to  be 
understood  in  terms  of  its  rules,  interface,  and  in  terms  of  the  concept  of  play.  The  term  ludology  arose 
within  the  context  of  non-electronic  games  and  board  games  in  particular  but  gained  popularity  after  it 
was  featured  in  a  1999  article  by  Gonzalo  Frasca  (a  game  designer  and  academic  researcher).  The 
name,  however,  has  not  yet  caught  on  fully.  Major  issues  in  the  field  concern  questions  of  narrative 
and  of  simulation  and  whether  video  games  are  either,  neither,  or  both. 

4  Simpson’s  paradox  (or  the  Yule-Simpson  effect)  is  a  statistical  paradox  wherein  the  successes  of 
groups  seem  reversed  when  the  groups  are  combined.  This  result  is  often  encountered  in  social  and 
medical  science  statistics  and  occurs  when  a  weighting  variable,  which  is  irrelevant  to  the  individual 
group  assessment,  must  be  used  in  the  combined  assessment. 
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This  is  an  area  in  which  many  people  are  interested  but  little  is  being  accom¬ 
plished.  Again,  closed-ended  computational  models,  the  workings  of  which  no  one  can 
explain  (e.g.,  Latent  Semantic  Analysis),  may  not  have  anything  to  do  with  reality. 

Why  not  start  with  an  anthropological  model  based  on  field  studies  of  what  peo¬ 
ple  actually  do,  how  they  describe  effectiveness,  and  so  forth? 

2.  Estimated  Costs 

A  $1-M  effort  to  study  the  differences  between  the  use  of  anthropological  vs. 
computer-science  models  would  be  fascinating. 
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APPENDIX  A 
GLOSSARY 


AAC 

augmentative  and  alternative  communication 

AAR 

after  action  review 

AMSC 

Anny  Management  Staff  College 

AoA 

analysis  of  alternatives 

AOT 

assignment-oriented  training 

ARFORGEN 

Army  Force  Generation 

ARI 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences 

AWC 

Anny  War  College 

BCKS 

Battle  Command  Knowledge  System 

CAL 

Center  for  Anny  Leadership 

CEU 

Continuing  Education  Unit 

CGSC 

Command  and  General  Staff  College 

COTS 

commercial  off-the-shelf 

DIME 

Diplomatic,  Information,  Military,  Economic 

DoD 

Department  of  Defense 

DOTMLPF 

Doctrine,  Organization,  Training,  Material,  Leadership,  Personnel, 
Facilities 

DVD 

digital  versatile  disc 

GEL 

Guided  Experiential  Learning 

HPI 

Human  Performance  Improvement 

IET 

Initial  Entry  Training 

IOC 

initial  operating  capability 

ISD 

Instructional  Systems  Design 

ITS 

Intelligent  Tutoring  System 

KM 

knowledge  management 

LMS 

Learning  Management  Systems 

MBA 

Master  of  Business  Administration 

MEC 

Mission  Essential  Capability 

METT-TC 

Mission,  Enemy,  Terrain  and  weather,  Troops  -  Time,  Civil 

MOS 

military  occupation  specialty 

NASA 

National  Aeronautics  and  Space  Administration 
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OAC 

Officer  Advanced  Course 

OOTW 

operations  other  than  war 

PECS 

picture  exchange  communication  system 

PoA 

plan  of  action 

POI 

program  of  instruction 

R&D 

research  and  development 

RECON 

Representing  Enriched  CONtext 

ROE 

rules  of  engagement 

ROI 

return  on  investment 

SAT 

System  Approach  to  Training 

SME 

subject  matter  expert 

TLAC 

Think  Like  A  Commander 

TRADOC 

U.S.  Army  Training  and  Doctrine  Command 
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CUMULATIVE  COSTS  AND  BENEFITS  TABLES 
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Table  B-1.  Cumulative  Y1  Costs  and  Benefits 


Short  Titles 

Cumulative 

Y1  Costs 
(K  Dollars) 

Cumulative 
Proportion  of 
Total  Benefit 

New  Training  R&D 

23,417 

1.000 

Folksonomies 

14,042 

0.988 

HPI  in  the  Army 

11,792 

0.978 

Training  Support  Tools 

1 1 ,292 

0.971 

Improvisation 

10,292 

0.947 

Critical  Thinking  Skills 

9,854 

0.934 

Leader  Socialization 

8,604 

0.891 

Predictive  Models 

8,063 

0.870 

Life-long  Learning 

7,313 

0.835 

Team  Task  Analysis 

6,313 

0.783 

The  GEL  Model 

5,313 

0.726 

R&D  on  dL 

5,063 

0.708 

Knowledge  Management 

4,563 

0.667 

Experience-Based  Learning 

3,563 

0.585 

Blended  Learning 

3,063 

0.537 

Training  Capabilities 

2,875 

0.518 

Diagnostic  Evaluation 

2,500 

0.474 

Lessons  Learned 

1,750 

0.380 

Implementation  Plan  for  dL 

1,250 

0.314 

Analyses  of  Training 

542 

0.180 

Tailored  Training 

250 

0.114 
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Table  B-2.  Cumulative  FYDP  Costs  and  Benefits 


Short  Titles 

Cumulative 
FYDP  Costs 
(K  Dollars) 

Cumulative 
Proportion  of 
Total  Benefit 

New  Training  R&D 

76,417 

1.000 

Folksonomies 

29,542 

0.988 

Improvisation 

21,292 

0.978 

HPI  in  the  Army 

20,104 

0.965 

Training  Support  Tools 

19,604 

0.958 

R&D  on  dL 

18,104 

0.934 

The  GEL  Model 

15,604 

0.893 

Life-long  Learning 

14,854 

0.875 

Critical  Thinking  Skills 

12,854 

0.823 

Leader  Socialization 

11,271 

0.780 

Knowledge  Management 

10,729 

0.760 

Lessons  Learned 

8,729 

0.677 

Predictive  Models 

7,229 

0.611 

Diagnostic  Evaluation 

6,479 

0.575 

Team  Task  Analysis 

4,729 

0.481 

Analyses  of  Training 

3,729 

0.425 

Training  Capabilities 

2,771 

0.358 

Experience-Based  Learning 

2,146 

0.314 

Blended  Learning 

1,563 

0.266 

Implementation  Plan  for  dL 

1,375 

0.248 

Tailored  Training 

500 

0.114 
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